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Abstract Given the ever increasing bandwidth of the 
visual sensory information available to autonomous a- 
gents and other automatic systems, it is becoming es- 
sential to endow them with a sense of what is worth- 
while their attention and what can be safely disregarded. 
This article presents a general mathematical framework 
to efficiently allocate the available computational re- 
sources to process the parts of the input that are rele- 
vant to solve a perceptual problem of interest. By solv- 
ing a perceptual problem we mean to find the hypoth- 
esis H (i.e., the state of the world) that maximizes 
a function L(H), referred to as the evidence, repre- 
senting how well each hypothesis "explains" the input. 
However, given the large bandwidth of the sensory in- 
put, fully evaluating the evidence for each hypothesis is 
computationally infcasible (e.g., because it would im- 
ply checking a large number of pixels). To address this 
problem we propose a mathematical framework with 
two key ingredients. The first one is a Bounding Mech- 
anism (BM) to compute lower and upper bounds of 
the evidence of a hypothesis, for a given computational 
budget. These bounds are much cheaper to compute 
than the evidence itself, can be refined at any time by 
increasing the budget allocated to a hypothesis, and 
are frequently sufficient to discard a hypothesis. The 
second ingredient is a Focus of Attention Mechanism 
(FoAM) to select which hypothesis' bounds should be 
refined next, with the goal of discarding non-optimal 
hypotheses with the least amount of computation. 



D. Rother • R. Vidal 
Johns Hopkins University 
Tel.: +1-410-516-6736 
E-mail: diroth@gmail.com 

S. Schiitz 

University of Gottingen 



The proposed framework has the following desirable 
characteristics: 1) it is very efficient since most hypothe- 
ses are discarded with minimal computation; 2) it is 
parallelizable; 3) it is guaranteed to find the globally 
optimal hypothesis or hypotheses; and 4) its running 
time depends on the problem at hand, not on the band- 
width of the input. In order to illustrate the general 
framework, in this article we instantiate it for the prob- 
lem of simultaneously estimating the class, pose and a 
noiseless version of a 2D shape in a 2D image. To do 
this, we develop a novel theory of semidiscrete shapes 
that allows us to compute the bounds required by the 
BM. We believe that the theory presented in this ar- 
ticle (i.e., the algorithmic paradigm and the theory of 
shapes) has multiple potential applications well beyond 
the application demonstrated in this article. 

Keywords Focus of Attention • shapes • shape priors • 
hypothesize-and-verify • coarse-to-fine • probabilistic 
inference • graphical models • image understanding 



1 Introduction 

Humans are extremely good at extracting information 
from images. They can recognize objects from many 
different classes, even objects they have never seen be- 
fore; they can estimate the relative size and position of 
objects in 3D, even from 2D images; and they can (in 
general) do this in poor lighting conditions, in cluttered 
scenes, or when the objects are partially occluded. How- 
ever, natural scenes are in general very complex (Fig. 
[T^l), full of objects with intricate shapes, and so rich 
in textures, shades and details, that extracting all this 
information would be computationally very expensive, 
even for humans. Experimental psychology studies, on 
the other hand, suggest that (mental) computation is 



2 




Fig. 1 (a) Natural scenes are so complex that it would be 
computationally wasteful for humans to extract all the infor- 
mation that they can extract from them, (b) Attention is task 
oriented: when focused on the task of counting the passes of 
a ball between the people in this video, many humans fail 
to see the gorilla in the midle of the frame (image from |37| . 
see details in that article), (c) Precision is task dependent: in 
order to pick up the glass, one does not need to estimate the 
locations of the mugs and the glass with the same precision. 
A rough estimate of the locations of the mugs is enough to 
avoid them; a better estimate is needed to pick up the glass. 

a limited resource that, when demanded by one task, 
is unavailable for another [14] . This is arguably why 
humans have evolved a focus of attention mechanism 
(FoAM) to discriminate between the information that 
is needed to achieve a specific goal, and the information 
that can be safely disregarded. 

Humans, for example, do not perceive every object 
that enters their field of view; rather they perceive only 
the objects that receive their focused attention (Fig.[lJ>) 
[37] . Moreover, in order to save computation, it is rea- 
sonable that even for objects that are indeed perceived, 
only the details that are relevant towards a specific goal 
are extracted (as beautifully illustrated in [5]). In par- 
ticular, it is reasonable to think that objects are not 
classified at a more concrete level than necessary (e.g., 
'terrier' vs. 'dog'), if this were more expensive than clas- 
sifying the object at a more abstract level, and if this 
provided the same amount of relevant information to- 
wards the goal [29] . Also we do not expect computation 
to be spent estimating other properties of an object 
(such as size or position) with higher precision than 
necessary if this were more expensive and provided the 
same amount of relevant information towards the goal 
(Fig#). 

1.1 Hypothesize- and-bound algorithms 

In this article we propose a mathematical framework 
that uses a FoAM to allocate the available computa- 
tional resources where they contribute the most to solve 
a given task. This mechanism is one of the parts of a 
novel family of inference algorithms that we refer to as 
hypothesize- and-bound (H&B) algorithms. These algo- 
rithms are based on the hypothesize- and-verify paradigm. 
In this paradigm a set of hypotheses and a function re- 
ferred to as the evidence are defined. Each hypothesis 
H represents a different state of the world (e.g., which 



objects are located where) and its evidence L(H) quan- 
tifies how well this hypothesis "explains" the input im- 
age. In a typical hypothesize-and-verify algorithm the 
evidence of each hypothesis is evaluated and the hy- 
pothesis (or group of hypotheses) with the highest evi- 
dence is selected as the optimal. 

However, since the number of hypotheses could be 
very large, it is essential to be able to evaluate the evi- 
dence of each hypothesis with the least amount of com- 
putation. For this purpose the second part of a H&B 
algorithm is a bounding mechanism (BM), which com- 
putes lower and upper bounds for the evidence of a hy- 
pothesis, instead of evaluating it exactly. These bounds 
are in general significantly less expensive to compute 
than the evidence itself, and they are obtained for a 
given computational budget (allocated by the FoAM), 
which in turn defines the tightness of the bounds (i.e., 
higher budgets result in tighter bounds). In some cases, 
these inexpensive bounds are already sufficient to dis- 
card a hypothesis (e.g., if the upper bound of L(Hi) is 
lower than the lower bound of L(i?2), Hi can be safely 
discarded). Otherwise, these bounds can be efficiently 
and progressively refined by spending extra computa- 
tional cycles on them (see Fig. [9]) . As mentioned above, 
the FoAM allocates the computational budget among 
the different hypotheses. This mechanism keeps track of 
the progress made (i.e., how much the bounds got closer 
to each other) for each computation cycle spent on each 
hypothesis, and decides on-the-fly where to spend new 
computation cycles in order to economically discard as 
many hypotheses as possible. Because computation is 
allocated where it is most needed, H&B algorithms are 
in general very efficient; and because hypotheses are 
discarded only when they are proved suboptimal, H&B 
algorithms are guaranteed to find the optimal solution. 



1.2 Application of H&B algorithms to vision 

The general inference framework mentioned in the pre- 
vious paragraphs is applicable to any problem in which 
bounds for the evidence can be inexpensively obtained 
for each hypothesis. Thus, to instantiate the framework 
to solve a particular problem, a specific BM has to be 
developed for that problem. (The FoAM, on the other 
hand, is common to many problems since it only com- 
municates with the BM by allocating the computational 
budget to the different hypotheses and by "reading" the 
resulting bounds.) 

In this article we illustrate the framework by instan- 
tiating it for the specific problem of jointly estimating 
the class and the pose of a 2D shape in a noisy 2D 
image, as well as recovering a noiseless version of this 
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shape. This problem is solved by "merging" informa- 
tion from the input image and from probabilistic mod- 
els known a priori for the shapes of different classes. 

As mentioned above, to instantiate the framework 
for this problem, we must define a mechanism to com- 
pute and refine bounds for the evidence of the hypothe- 
ses. To do this, we will introduce a novel theory of 
shapes and shape priors that will allow us to efficiently 
compute and refine these bounds. While still of practi- 
cal interest, we believe that the problem chosen is sim- 
ple enough to best illustrate the main characteristics 
of H&B algorithms and the proposed theory of shapes, 
without occluding its main ideas. Another instantiation 
of H&B algorithms, which we describe in [35], tackles 
the more complex problem of simultaneous object clas- 
sification, pose estimation, and 3D reconstruction, from 
a single 2D image. In that work we also use the theory 
of shapes and shape priors to be described in this article 
to construct the BM for that problem. 

1.3 Paper contributions 

The framework we propose has several novel contri- 
butions that we group in two main areas, namely: 1) 
the inference framework using H&B algorithms; and 2) 
the shape representations, priors, and the theory devel- 
oped around them. The following paragraphs summa- 
rize these contributions, while in the next section we 
put them in the context of prior work. 

The first contribution of this paper is the use of 
H&B algorithms for inference in probabilistic graphi- 
cal models. In particular, inference in graphical mod- 
els containing loops. This inference method is not gen- 
eral, i.e., it is not applicable to any directed graph with 
loops. Rather it is specifically designed for the kinds of 
graphs containing pixels and voxels that are often used 
for vision tasks (which typically have a large number 
of variables and a huge number of loops among these 
variables). The proposed inference framework has sev- 
eral desirable characteristics. First, it is general, in the 
sense that the only requirement for its application is a 
BM to compute and refine bounds for the evidence of 
a hypothesis. Second, the framework is computationally 
very efficient, because it allocates computation dynam- 
ically where it is needed (i.e., refining the most promis- 
ing hypotheses and examining the most informative im- 
age regions). Third, the total computation does not de- 
pend on the arbitrary resolution of the input image, 
but rather on the task at hand, or more precisely, on 
the similarity between the hypotheses that are to be 
distinguished. (In other words, "easy" tasks are solved 
very fast, while only "difficult" tasks require process- 
ing the image completely.) This allows us to avoid the 



common preprocessing step of downsampling the input 
image to the maximal resolution that the algorithm can 
handle, and permits us to use the original (possibly very 
high) resolution only in the parts of the image where it 
is needed. Fourth, the framework is fully parallelizable, 
which allows it to take advantage of GPUs or other par- 
allel architectures. And fifth, it is guaranteed to find the 
globally optimal solution (i.e., the hypothesis with the 
maximum evidence), if it exists, or a set of hypotheses 
that can be formally proved to be undistinguishable at 
the maximum available image resolution. This guaran- 
tee is particularly attractive when the subjacent graph- 
ical model contains many loops, since existing proba- 
bilistic inference methods are either very inefficient or 
not guaranteed to find the optimal solution. 

The second contribution relates to the novel shape 
representations proposed, and the priors presented to 
encode the shape knowledge of the different object cla- 
sses. These shape representations and priors have three 
distinctive characteristics. First, they are able to rep- 
resent a shape with multiple levels of detail. The level 
of detail, in turn, defines the amount of computation 
required to process a shape, which is critical in our 
framework. Second, it is straightforward and efficient 
to project a 3D shape expressed in these representa- 
tions to the 2D image plane. This will be essential in 
the second part of this work [32] to efficiently compute 
how well a given 3D reconstruction "explains" the in- 
put image. And third, based on the theory developed 
for these shape representations and priors, it is possible 
to efficiently compute tight log-probability bounds. More- 
over, the tightness of these bounds also depends on the 
level of detail selected, allowing us to dynamically trade 
computation for bound accuracy. In addition, the the- 
ory introduced is general and could be applied to solve 
many other problems as well. 

1.4 Paper organization 

The remainder of this paper is organized as follows. Sec- 
tion [2] places the current work in the context of prior 
relevant work, discussing important connections. Sec- 
tion [3] describes the proposed FoAM. To illustrate the 
application of the framework to the concrete problem 
of 2D shape classification, denoising, and pose estima- 
tion, in Section [4] we formally define this problem, and 
in Section [6] we develop the BM for it. In order to de- 
velop the BM, we first introduce in Section[5]a theory of 
shapes and shape priors necessary to compute the de- 
sired bounds for the evidence. Because the FoAM and 
the theory of shapes described in section [3] and [5] re- 
spectively, are general (i.e., not only limited to solve 
the problem described in Section[4|, these sections were 
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written to be self contained. Section [7] presents experi- 
mental results obtained with the proposed framework, 
and Section [8] concludes with a discussion of the key 
contributions and directions for future research. In the 
continuation of this work |32j , we extend the theory pre- 
sented in this article to deal with a more complex prob- 
lem involving not just 2D shapes, but also 3D shapes 
and their 2D projections. 

2 Prior work 

As mentioned in the previous section, this article pre- 
sents contributions in two main areas: 1) the inference 
framework based on the FoAM; and 2) the shape repre- 
sentations proposed and the theory developed around 
them. For this reason, in this section we briefly review 
prior related work in these areas. 

2.1 FoAM 

Many computational approaches that rely on a focus 
of attention mechanism have been proposed over the 
years, in particular to interpret visual stimuli. These 
computational approaches can be roughly classified into 
two groups, depending on whether they are biologically 
inspired or not. 

Biologically inspired approaches 9,39,36 , by defi- 
nition, exploit characteristics of a model proposed to 
describe a biological system. The goals of these ap- 
proaches often include: 1) to validate a model proposed 
for a biological system; 2) to attain the outstanding 
performance of biological systems by exploiting some 
characteristics of these systems; and 3) to facilitate the 
interaction between humans and a robot by emulat- 
ing mechanisms that humans use (e.g., joint attention 
[15]). Biological strategies, though optimized for eyes 
and brains during millions of years of evolution, are 
not necessarily optimal for current cameras and com- 
puter architectures. Moreover, since the biological at- 
tentional strategies are adopted at the foundations of 
these approaches by fiat (instead of emerging as the so- 
lution to a formally defined problem) , it is often difficult 
to rigorously analyze the optimality of these strategies. 
In addition, these attentional strategies were in general 
empirically discovered for a particular sensory modality 
(predominantly vision) and are not directly applicable 
to other sensory modalities. Moreover, these strategies 
are not general enough to handle simultaneous stim- 
uli coming from several sensory modalities (with some 
exceptions, e.g., [I]). 

Since in this article we are mainly interested in im- 
proving the performance of a perceptual system (possi- 



bly spanning several sensory modalities), and since we 
want to be able to obtain optimality guarantees, we do 
not focus further on biologically inspired approaches. 

The second class of focus of attention mechanisms 
contains those approaches that are not biologically in- 
spired. Within this class we focus on those approaches 
that are not ad hoc (i.e., they are rigorously derived 
from first principles) and are general (i.e., they are able 
to handle different sensory modalities and tasks). This 
subclass contains at least two other approaches (apart 
from ours): Branch and Bound (B&B) and Entropy 
Pursuit (EP). 

In a B&B algorithm 4) , as in a H&B algorithm, an 
objective function is defined over the hypothesis space 
and the goal of the algorithm is to select the hypothesis 
that maximizes this function. A B&B algorithm pro- 
ceeds by dividing the hypothesis space into subspaces, 
computing bounds of the objective function for each 
subspace (rather than for each hypothesis in the sub- 
space), and discarding subspaces that can be proved 
to be non-optimal (because their upper bound is lower 
than that of some other subspace) . In these algorithms 
computation is saved by evaluating whole groups of hy- 
potheses, instead of evaluating each individual hypoth- 
esis. In contrast, in our approach the hypothesis space 
is discrete and bounds are computed for every hypoth- 
esis in the space. In this case computation is saved 
by discarding most of these hypotheses with very lit- 
tle computation. In other words, for most hypotheses 
the inexpensive bounds computed for the evidence of 
these hypotheses are enough to discard these hypothe- 
ses. As will be discussed in Section [5J this approach is 
complementary to B&B, and it would be beneficial to 
integrate both approaches into a single framework. Due 
to space limitations, however, this is not addressed in 
this article. 

In an EP algorithm 11,38J a probability distribu- 
tion is defined over the hypothesis space. Then, during 
each iteration of the algorithm, a test is performed on 
the input, and the probability distribution is updated 
by taking into account the result of the test. This test 
is selected as the one that is expected to reduce the 
entropy of the distribution the most. The algorithm 
terminates when the entropy of the distribution falls 
below a certain threshold. A major difference between 
EP and B&B/H&B algorithms is that in each iteration 
of EP a test is selected and the probability of each (of 
potentially too many) hypothesis is updated. In con- 
trast, in each iteration of B&B/H&B, one hypothesis 
( or one group of hypotheses ) is selected and only the 
bounds corresponding to it are updated. Unlike B&B 
and H&B algorithms, EP algorithms are not guaran- 
teed to find the optimal solution. 



5 



A second useful criterion to classify computational 
approaches that rely on a FoAM considers whether at- 
tention is controlled only by "bottom-up" signals de- 
rived from salient stimuli, or whether it is also con- 
trolled by "top-down" signals derived from task de- 
mands, or from what a model predicts to be most rele- 
vant. Bottom-up approaches (e.g., |17j ) are also known 
as data-driven approaches, while top-down approaches 
(e.g., |24| ) are also known as task-driven approaches. 
Even though the significance of top-down signals in 
biological systems is well known, most current com- 
puter systems only consider bottom- up signals [5]. In 
contrast, all the three algorithmic paradigms described 
(H&B, B&B and EP), depending on the specific instan- 
tiation of these paradigms, are able to handle bottom- 
up as well as top-down signals. In particular, in the 
instantiation of H&B algorithms presented in Section 
[IJ both kinds of signals are considered (in fact, it will 
be seen in Equation ( 39 1 that they play a symmetric 



role) . In addition, in all the three algorithmic paradigms 
described above there is an explicit FoAM to control 
where the computation is allocated. 

2.2 Inference framework 

Many methods have been proposed to perform inference 
in graphical models |18j . Message passing algorithms 
are one class of these methods. Belief propagation (BP) 
is an algorithm in this class that is guaranteed to find 
the optimal solution in a loopless graph (i.e., a polytree) 
P]. The loopy belief propagation (LBP) and the junc- 
tion tree (JT) algorithms are two algorithms that ex- 
tend the capabilities of the basic BP algorithm to han- 
dle graphs with loops. In LBP messages are exchanged 
exactly as in BP, but multiple iterations of the basic 
BP algorithm are required to converge to a solution. 
Moreover, the method is not guaranteed to converge to 
the optimal solution in every graph but only in some 
types of graphs [42]. In the JT algorithm [19], BP is 
run on a modified graph whose cycles have been elimi- 
nated. To construct this modified graph, the first step is 
"moralization," which consists of marrying the parents 
of all the nodes. For the kinds of graphs we are inter- 
ested in, however, this dramatically increases the clique 
size. While the JT algorithm is guaranteed to find the 
optimal solution, in our case this algorithm is not effi- 
cient because its complexity grows exponentially with 
the size of the largest clique in the modified graph. 

Two standard "tricks" to perform exact inference 
(using BP) by eliminating the loops of a general graph 
are: 1) to merge nodes in the graph into a "supernode," 
and 2) to make assumptions about the values of (i.e., to 
instantiate) certain variables, creating a different graph 



for each possible value of the instantiated variables (i.e., 
for each hypothesis) [25]. These approaches, however, 
bring their own difficulties. On the one hand, merging 
nodes results in a supernode whose number of states 
is the product of the number of states of the merged 
nodes. On the other hand, instantiating variables forces 
us to solve an inference problem for a potentially very 
large number of hypotheses. 

In this work we propose a different approach to 
merge nodes that does not run into the problems men- 
tioned above. Specifically, instead of assuming that the 
image domain is composed of a finite number of discrete 
pixels and merging them into supernodes, we assume 
that the image domain is continuous and consists of an 
infinite number of "pixels." We then compute "sum- 
maries" of the values of the pixels in each region of 
the domain (this is formally described in Section [5]). In 
order to solve the inference efficiently for each hypoth- 
esis, the total computation per hypothesis is trimmed 
down by using lower and upper bounds and a FoAM, 
as mentioned in Section [T] 

2.3 Shape representations and priors 

Since shape representations and priors are such essen- 
tial parts of many vision systems, over the years many 
shape representations and priors have been proposed 
(see reviews in [HE]). Among these, only a small frac- 
tion have the three properties required by our system 
and mentioned in Section [l.3| i.e., support multiple lev- 
els of detail, efficient projection, and efficient computa- 
tion of bounds. 

Scale space and orthonormal basis representations 
have the property that they can encode multiple lev- 
els of detail. In the scale-space representation |20j . a 
shape (or image in general) is represented as a one- 
parameter family of smoothed shapes, parameterized 
by the size of the smoothing kernel used for suppress- 
ing fine-scale structures. Therefore, the representation 
contains a smoothed copy of the original shape at each 
level of detail. In the orthonormal basis representation, 
on the other hand, a shape is represented by its co- 
efficients in an orthonormal basis. To compute these 
coefficients the shape is first expressed in the same rep- 
resentation as the orthonormal basis. For example, in 
[23] and [26] the contour of a shape is expressed in 
spherical wavelets and Fourier bases, respectively, and 
in |34j the signed distance function of a shape is writ- 
ten in terms of the principal components of the signed 
distance functions of shapes in the training database. 
The level of detail in this case is defined by the num- 
ber of coefficients used to represent the shape in the 
basis. While these shape representations have the first 
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property mentioned above (i.e., multiple levels of de- 
tail), they do not have the other two, that is that it 
is not trivial to efficiently project 3D shapes expressed 
in these representations to the 2D image plane, or to 
compute the bounds that we want to compute. 

The shape representations we propose, referred to 
as discrete and semidiscrete shape representations (de- 
fined in Section [5] and shown in Fig. [5| are respec- 
tively closer to region quadtrees/octrees [S3] and to 
occupancy grids [7]. In fact, the discrete shape rep- 
resentation we propose is a special case of a region 
quadtree/ octree in which the rule to split an element is 
a complex function of the input data, the prior knowl- 
edge, and the interaction with other hypotheses. Quad- 
trees and octrees have been previously used for 3D 
recognition [5] and 3D reconstruction |57j from mul- 
tiple silhouettes (not from a single one, to the best of 
our knowledge, as we do in 32J). Occupancy grids, on 
the other hand, are significantly different from semidis- 
crete shapes since they store at each cell a qualitatively 
different quantity: occupancy grids store the posterior 
probability that an object is in the cell, while semidis- 
crete shapes store the measure of the object in the cell. 

3 Focus of attention mechanism 

In Section[l]we mentioned that a hypothesize- and-bound 
(H&B) algorithm has two parts: 1) a focus of attention 
mechanism (FoAM) to allocate the available computa- 
tion cycles among the different hypotheses; and 2) a 
bounding mechanism (BM) to compute and refine the 
bounds of each hypothesis. In this section we describe 
in detail the first of these two parts, the FoAM. 

Let / be some input and let H = {Hi, . . . , Hn h } 
be a set of Njj hypotheses proposed to "explain" this 
input. In our problem of interest (formally described in 
Section [4]) the input / is an image, and each of the hy- 
potheses corresponds to the 2D pose and class of a 2D 
shape in this input image. However, from the point of 
view of the FoAM, it is not important what the input 
actually is, or what the hypotheses actually represent. 
The input can be simply thought of as "some informa- 
tion about the world acquired through some sensors," 
and the hypotheses can be simply thought of as repre- 
senting a "possible state of the world." 

Suppose that there exist a function L(H) that quan- 
tifies the evidence in the input / supporting the hy- 
pothesis H. In Section [4] the evidence for our problem 
is shown to be related to the log-joint probability of 
the image / and the hypothesis H. But again, from the 
point of view of the FoAM, it is not important how this 
function is defined; it only matters that hypotheses that 
"explain" the input better produce higher values. Thus, 



part of the goal of the FoAM is to select the hypothe- 
sis (or group of hypotheses) Hi* that best explain the 
input image, i.e., 

H^ = argmaxL(i/). (1) 

Heu 

Now, suppose that the evidence L{Hi) of a hypoth- 
esis Hi is very costly to evaluate (e.g., because a large 
number of pixels must be processed to compute it), but 
lower and upper bounds for it, L(Hi) and L(Hi), respec- 
tively, can be cheaply computed by the BM. Moreover, 
suppose that the BM can efficiently refine the bounds 
of a hypothesis Hi if additional computational cycles 
(defined below) are allocated to the hypothesis. Let us 
denote by L n . (Hi) and L n . (Hi), respectively, the lower 
and upper bounds obtained for L(H{) after ni compu- 
tational cycles have been spent on Hi. If the BM is well 
defined, the bounds it produces must satisfy 

L n . +1 (Hi) > L nz (H t ), and L 7H+1 (H t ) <L 7H (H t ), (2) 

for every hypothesis H iy and every > (assume 
that ni — is the initialization cycle in which the 
bounds are first computed). In other words, the bounds 
must not become looser as more computational cycles 
are invested in their computation. Note that we ex- 
pect different numbers of cycles to be spent on differ- 
ent hypotheses, ideally with "bad" hypotheses being 
discarded earlier than "better" ones (i.e., n\ < n 2 if 
L(H X ) « L(H 2 )). 

The "computational cycles" mentioned above are 
our unit to measure the computational resources spent. 
Each computational cycle, or just cycle, is the compu- 
tation that the BM spends to refine the bounds. While 
the exact conversion rate between cycles and operations 
depends on the particular BM used, what is important 
from the point of view of the FoAM is that all refine- 
ment cycles take approximately the same number of 
operations (defined to be equal to one computational 
cycle) . 

The full goal of the FoAM can now be stated as to 
select the hypothesis Hi that satisfies 

L ni (H i )>L nj (H j ) Vj^i, (3) 

while minimizing the total number of cycles spent, Ylj=i 
nj. If these inequalities are satisfied, it can be proved 
that Hi is the optimal hypothesis, without having to 
compute exactly the evidence for every hypothesis (which 
is assumed to be much more expensive than just com- 
puting the bounds). However, it is possible that after 
all the hypotheses in a set i; C i have been refined 
to the fullest extent possible, their upper bounds are 
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still bigger than or equal to the maximum lower bound 
7 = maxi/ iSHi L (Hi), i.e., 



(4) 



In this situation all the hypotheses in Hi could possi- 
bly be optimal, but we cannot say which one actually 
is. We just do not have the right input to distinguish 
between them (e.g., because the resolution of the input 
image is insufficient). We say that these hypotheses are 
indistinguishable given the current input. In short, the 
FoAM will terminate either because it has found the op- 
timal hypothesis (satisfying ([3])), or because it has found 
a set of hypotheses that are indistinguishable from the 
optimal hypothesis given the current input (and satisfies 

These termination conditions can be achieved by 
very different budgets that allocate different number of 
cycles to each hypothesis. We are interested in finding 
the budget that achieves them in the minimum number 
of cycles. Finding this minimum is in general not pos- 
sible since the FoAM does not "know," a priori, how 
the bounds will change for each cycle it allocates to a 
hypothesis. For this reason, we propose a heuristic to se- 
lect the next hypothesis to refine at each point in time. 
Once a hypothesis is selected, one cycle is allocated to 
this hypothesis, which is thus refined once by the BM. 
This selection-refinement cycle is continued until termi- 
nation conditions are reached. 

According to the heuristic proposed, the next hy- 
pothesis to refine, H^, is chosen as the one that is ex- 
pected to produce the greatest reduction AP(H^) in 
the following potential P, 
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(5) 



where A is the set of all the hypotheses not yet dis- 
carded (i.e., those that are active), 7 is the maximum 
lower bound defined before, and is the number of re- 
finement cycles spent on hypothesis Hi . This particular 
expression for the potential was chosen for two reasons: 
1) because it reflects the workload left to be done by the 
FoAM; and 2) because it is minimal when termination 
conditions have been achieved. 

In order to estimate the potential reduction AP(H) 
expected when hypothesis H is refined (as required by 
the heuristic), we need to first define a few quantities 
(Fig. [2]). We define the margin M n (H) of a hypothesis 
H after n cycles have been spent on it, as the difference 
between its bounds, i.e., M n (H) = L n (H) — L n (H). 
Then we define the reduction of the margin of this 
hypothesis during its n-th refinement as AM n (H) = 
M n ^i(H) — M n (H) . It can be seen that this quantity is 
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Fig. 2 (a) Bounds for the evidence L(Hi) of three active 
hypotheses (Hi, H2, and i/3) after refinement cycle t. (b-c) 
Bounds for the same three hypotheses after refinement cycle 
t+1, assumming that either Hi (b) or H2 (c) was refined dur- 
ing cycle t+1. Each rectangle represents the interval where 
the evidence of a hypothesis is known to be. Green and red 
rectangles denote active and discarded hypotheses, respec- 
tively. The maximum lower bound in each case (y a , 7b, and 
7c) is represented by the black dashed line. It can be seen 
that A~/(Hi) = 7b - 7a < 7c - 7a = Aj(H 2 ). The po- 
tential in each case (P a , Pb, and P c ) is represented by the 
sum of the gray parts of the intervals. It can be seen that 
AP{Hi) = P a -P b <Pa-P c = AP{H 2 ). 

positive, and because in general early refinements pro- 
duce larger margin reductions than later refinements, 
it has a general tendency to decrease. Using this quan- 
tity we predict the reduction of the margin in the next 
refinement using an exponentially weighted moving av- 
erage, AM n+l {H) 4 a AM n (H) + (1 - a)AM n (H), 
where < a < 1. This moving average is initialized as 
AM (H) = (3M (H). (In this work we used a = 0.9 
and [3 = 0.25.) 

The predicted potential reduction AP(H) depends 
on whether the refinement of H is expected to increase 
7 or not: when 7 increases, every term in Q is re- 
duced; when it does not, only the term corresponding 
to H is reduced (compare figures^ and |2j;). Let us as- 
sume that the reduction in the upper bound, L n (H) — 
L n+ i(H), is equal to the increase in the lower bound, 
L n+ i(H) — L n (H), which is therefore predicted to be 
equal to AM n+ x(H)/2. Let us define A^(H) to be 
the increase in 7 when H is refined, thus Aj(H) = 
max{L„(iJ) + AM n+1 {H)/2 - 7,0}. Then the follow- 
ing expression is an estimate for the potential reduction 
when H is refined, 



AP(H) 



AM n+1 (H)/2 
AM n+1 {H)/2, 



|A|Z\ 7 (iJ), if Ay(ff) > 
otherwise. 

(6) 



As mentioned before, the hypothesis that maximizes 
this quantity is the one selected to be refined next. 

The algorithm used by the FoAM is thus the fol- 
lowing (a detailed explanation is provided immediately 
afterwards): 



7 ' 

for i 



-00 



ii = l to Njj do 

[L(Hi),L(Hi)} <- ComputeBounds(i2' i 
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4: if L(Hi) > 7 then 

5: 7 <" 

6: end if 

7: if L(fli) > 7 then 

8: AM {Hi) ^/3M (Hi) 

9: Compute AP(Hi) 

10: AInsert(#i, AP(i^)) 

11: end if 

12: end for 

13: while not reached Termination Conditions do 

14: flj 4- AGetMax() 

15: if L(Hi) > 7 then 

16: [L(Hi),L(Hi)] <- RefineBounds(Fi) 

17: Compute AP(Hi) 

18: A Insert ( i/j, AP(i/,)) 

19: if > 7 then 

20: 7 4- L(i^) 

21: end if 

22: end if 

23: end while 

The first stage of the algorithm is to initialize the 
bounds for all the hypotheses (line 3 above), use these 
bounds to estimate the expected potential reduction for 
each hypothesis (lines 8-9), and to insert the hypotheses 
in the priority queue A using the potential reduction as 
the key (line 10). This priority queue A, supporting the 
usual operations Insert and GetMax, contains the hy- 
potheses that are active at any given time. The GetMax 
operation, in particular, is used to efficiently find the 
hypothesis that, if refined, is expected to produce the 
greatest potential reduction. During this first stage the 
maximum lower bound 7 is also initialized (lines 1 and 
4-6). 

In the second stage of the algorithm, hypotheses are 
selected and refined alternately until termination con- 
ditions are reached. The next hypothesis to refine is 
simply obtained by extracting from the priority queue 
A the hypothesis that is expected to produce, if re- 
fined, the greatest potential reduction (line 14). If this 
hypothesis is still viable (line 15), its bounds are re- 
fined (line 16), its expected potential reduction is re- 
computed (line 17), and it is reinserted into the queue 
(line 18). If necessary, the maximum lower bound is 
also updated (lines 19-21). One issue to note in this 
procedure is that the potential reductions used as key 
when hypotheses are inserted in the queue A are out- 
dated once 7 is modified. Nevertheless this approxima- 
tion works well in practice and allows a complete hy- 
pothesis selection/refinement cycle (lines 13-23) to run 
in 0(log I A|) where |Aj is the number of active hypothe- 
ses. This complexity is determined by the operations on 
the priority queue. 



Rother and Sapiro [3T] have suggested a different 
heuristic to select the next hypothesis to refine. Their 
heuristic consists on selecting the hypothesis whose cur- 
rent upper bound is greatest. This heuristic, however, in 
general required more cycles than the heuristic we are 
proposing here. To see why, consider the case in which, 
after some refinement cycles, two active hypotheses H\ 
and H 2 still remain. Suppose that Hi is better than 
H 2 (L ni (Hi) ^> L n2 (H 2 )), but at the same time it has 
been more refined {n\ n 2 ). As mentioned before, be- 
cause of the decreasing nature of AM, in these condi- 
tions we expect AM n2 (H 2 ) > AM ni (Hi). Therefore, 
if we chose to refine Hi (as in [3T]) many more cycles 
will be necessary to distinguish between the hypotheses 
than if we had chosen to refine H 2 (as in the heuris- 
tic explained above). However, the strategy of choosing 
the less promising hypothesis is only worthwhile when 
there are few hypotheses remaining, since computation 
in that case is invested in a hypothesis that is ultimately 
discarded. The desired behavior is simply and automat- 
ically obtained by minimizing the potential defined in 
([5]) , and this ensures that computation is spent sensibly. 

4 Definition of the Problem 

The FoAM described in the previous section is a gen- 
eral algorithm that can be used to solve many different 
problems, as long as: 1) the problems can be formu- 
lated as selecting the hypothesis that maximizes some 
evidence function within a set of hypotheses, and 2) 
a suitable BM can be defined to bound this evidence 
function. To illustrate the use of the FoAM to solve 
a concrete problem, in this section we define the prob- 
lem, and in Section [6] we derive a BM for this particular 
problem. 

Given an input image / : Q — > M. c (c € N, c > 0) in 
which there is a single "shape" corrupted by noise, the 
problem is to estimate the class K of the shape, its pose 
T, and recover a noiseless version of the shape. This 
problem arises, for example, in the context of optical 
character recognition [TU] and shape matching [40 j . For 
clarity it is assumed in this section that fl C Z 2 (i.e., 
the image is composed of discrete pixels arranged in a 
2D grid). 

To solve this problem using a H&B algorithm, we 
define one hypothesis H for every possible pair (K,T). 
By selecting a hypothesis, the algorithm is thus estimat- 
ing the class K and pose T of the shape in the image. 
As we will later show, in the process a noiseless version 
of the shape will also be obtained. 

In order to define the problem more formally, sup- 
pose that the image domain fl contains n pixels, Xi , . . . , 
x„, and that there are Nk distinct possible shape classes, 
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each one characterized by a known shape prior Bk (1 < 
K < Nk) defined on the whole discrete plane Z 2 , also 
containing discrete pixels. Each shape prior Bk speci- 
fies, for each pixel x' £ Z 2 , the probability that the pixel 
belongs to the shape q, pb k {x') — P(q(x') = l\K), or 
to the complement of the shape, P(q(x!) = 0\K) 
I — p Bl( (x / ). We assume that pb k is zero everywhere, 
except (possibly) in a region £2k C Z 2 called the sup- 
port of pb k - We will say that a pixel x' belongs to 
the Foreground if q(x') = 1, and to the Background if 
g(x') = (Foreground and Background are the labels 
of the two possible states of a shape for each pixel). 

Let T £ {Ti , . . . , T/v T } be an affinc transforma- 
tion in K 2 , and call B H (recall that H = (K,T)) the 
shape prior that results from transforming Bk by T, 
i.e., PBh (x) 4 P( 9 ( x ) = 4 ps^T^x) (disre- 

gard for the moment the complications produced by 
the misalignment of pixels). The state q(x) in a pixel 
x is thus assumed to depend only on the class K and 
the transformation T (in other words, it is assumed to 
be conditionally independent of the states in the other 
pixels, given the hypothesis H). 

Now, suppose that the shape q is not observed di- 
rectly, but rather that it defines the distribution of a 
feature (e.g., colors, edges, or in general any feature) 
to be observed at a pixel. In other words, if a pixel x 
belongs to the background (i.e., if g(x) = 0), its feature 
f(x) is distributed according to the probability den- 
sity function p x (/(x)|g(x) = 0), while if it belongs to 
the foreground (i.e., if g(x) = 1), /(x) is distributed 
according to p x (/(x)|g(x) = 1) (the subscript x in p x 
was added to emphasize the fact that the probability 
of observing a feature /(x) at a pixel x depends on the 
state of the pixel q(x) and on the particular pixel x, 
or in other terms, p x (/oko) ¥= PyCfoko) if x ^ y and 
/o and qo are two arbitrary values of / and q, respec- 
tively). This feature /(x) is assumed to be independent 
of the feature /(y) and the state q(y) in every other 
pixel y, given q(x). 

The conditional independence assumptions descri- 
bed above can be summarized in the factor graph of 
Fig. [| (see for more details on factor graphs). It then 
follows that the joint probability of all pixel features /, 
all states q, and the hypothesis H — (K,T), is 

p(f,q,H) = P(H) J] P x (/(x)|g(x))P(g(x)|ff). (7) 
men 

Then, our goal can be simply stated as solving 



K 



1- Object Class 



^3 






and Pose 




\ / • 






\ / 




2- Shape prior 


(?oo) 


3- 2D 

Segmentation 








4- Feature model 




5- Pixel features 



m8uxp(f.q,H) = maxL"(ff), with 

q,H H£U 

L"(H)^m a xp(f,q,H). 



(8) 
(9) 



Fig. 3 Factor graph proposed to solve our problem of inter- 
est. A factor graph, [2], has a variable node (circle) for each 
variable, and a factor node (square) for each factor in the 
system's joint probability. Factor nodes are connected to the 
variable nodes of the variables in the factor. Observed vari- 
ables are shaded. A plate indicates that there is an instance 
of the nodes in the plate for each element in a set (indicated 
on the lower right). The plate in this graph hides the existing 
loops. See text for details. 

We could solve this problem naively by computing ([9| 
for every H € H. However, to compute (|9| all the pixels 
in the image (or at least all the pixels in the support 
of Qk) need to be processed in order to evaluate the 
product in ^ (since the solution q* that maximizes 
([9| can be written explicitly). Because this might be 
very expensive, we need a BM to evaluate the evidence 
without having to process every pixel in the image. 

Therefore, instead of using this naive approach, we 
will use an H&B algorithm to find the hypothesis H 
that maximizes an expression simpler than L" '(H), that 
is equivalent to it (in the sense that it has the same max- 
ima) . This simpler expression is what we have called the 
evidence, L(H), and will be derived from ([9| in Section 
6.1| Before deriving the evidence, however, in Section 
5] we present the mathematical framework that will al- 
low us to do that, and later to develop the BM for this 
evidence. 



5 A new theory of shapes 

As mentioned before, H&B algorithms have two parts: 
a FoAM and a BM. The FoAM was already introduced 
m Section El While the same FoAM can be used to 
solve many different problems, each BM is specific to 
a particular problem. Towards defining the BM for the 
problem described in the previous section (that will be 
done in Section [6]), in this section we introduce a math- 
ematical framework that will allow us to compute the 
bounds. 

To derive formulas to bound the evidence L(H) of a 
hypothesis H (the goal of the BM) for our specific prob- 
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n 1 



Fig. 4 The set Q and three possible partitions of it. Each 
square represents a partition element. 77 2 is finer than 77 1 
(ft 2 < II 1 ). II 3 is not comparable with neither iJ 1 nor with 

n 2 . 

lem of interest, we introduce in this section a framework 
to represent shapes and to compute bounds for their 



log-probability (in Section 6.1 we will show that this 
log-probability is closely related to the evidence) . Three 
different shape representations will be introduced (Fig. 
[5j. Continuous shapes are the shapes that we would 
observe if our cameras (or 3D scanners) had "infinite 
resolution." In that case it would be possible to com- 
pute the evidence 1(H) of a hypothesis with "infinite 
precision" and therefore always select the (single) hy- 
pothesis whose evidence is maximum (except in con- 
cocted examples which have very low probability of oc- 
curring in practice). However, since real cameras and 
scanners have finite resolution, we introduce two other 
shape representations that are especially suited for this 
case: discrete and semidiscrete shape representations. 
Discrete shapes will allow us to compute a lower bound 
1(H) for the evidence of a hypothesis H. Semidiscrete 
shapes, on the other hand, will allow us to compute 
an upper bound L(H) for the evidence of a hypothe- 
sis H . Discrete and semidiscrete shapes arc defined on 
partitions of the input image (i.e., non-overlapping re- 
gions that completely cover the image, see Fig.j4|. Finer 
partitions result in tighter bounds, more computation, 
and the possibility of distinguishing more similar hy- 
potheses. Coarser partitions on the other hand, result 
in looser bounds, less computation and more hypothe- 
ses that are indistinguishable (in the partition). 

Previously we have assumed that the image domain, 
Q C Z 2 , consisted on discrete pixels arranged in a 2D 
grid. For reasons that will soon become clear, however, 
we assume from now on that the image domain Q C 
M 2 is continuous. Thus, to discretize this continuouos 
domain we rely on "partitions," defined next. 

Definition 1 (Partitions) Given a set Q C M. d , a par- 
tition n(il) = {]7i, . . . , fl n } with Qi 7^ 0, is a disjoint 
cover of the set Q (Fig. [4]). Formally, H(fl) satisfies 



{Jfh 



Q, and J?,- D fit 



Vi ^ j. 



(10) 



A partition H \Q) = {Qi, . . . , J? n } is said to be uni- 
form if all the elements in the partition have the same 
measure \ fii\ = ^ for i = 1, . . . ,n. (Throughout this 




Fig. 5 The three different shape representations used in this 
article: the continuous shape S (a), the discrete shape S (b), 
and the semidiscrete shape S (c). S and S are two approxi- 
mations of S in a finite partition (indicated by the red lines). 
The gray levels in S represents how much of an element is 
full. See text for details. 

article we use the notation |J?| to refer to, depending on 
the context, the measure or the cardinality of a set fl.) 
This measure is referred to as the unit size of the parti- 
tion. For d = 2 and d = 3, we will refer to the elements 
of the partition (fii) as pixels and voxels, respectively. 

Given two partitions 77 1 (f2) and H 2 (12) of a set 
Q C R d , II 2 is said to be finer than II 1 , and II 1 is 
said to be coarser than II 2 , if every element of H 2 is 
a subset of some element of 77 1 (Fig. [4|. We denote 
this relationship as H 2 < II 1 . Note that two partitions 
are not always comparable, thus the binary relationship 
"<" defines a partial order in the space of all partitions. 



5.1 Discrete shapes 

Definition 2 (Discrete shapes) Given a partition 
II(Q) of the set Q C R d , the discrete shape S (Fig. 
[^d) is defined as the function S : II(Q) -» {0, 1}. 

Definition 3 (Log-probability of a discrete shape) 

Let S be a discrete shape in some partition 7T(i7) = 
{I2\, . . . , and let B = {B\, . . . , B n } be a family of 
independent Bernoulli random variables referred to as 
a discrete Bernoulli field (BF). Let B be characterized 
by the success rates p§{i) = P{Bi = 1) G (e, 1 — e) for 
i = 1, . . . , n, and < e <C 1. To avoid the problems 
derived from assuming complete certainty, i.e. success 
rates of or 1, following Cromwell's rule [5T], we will 
only consider success rates in the open interval (e, 1— e). 
The log-probability of a discrete shape is defined as 



log P(B = S) = logP ( Bi = S(Qi) 



i=l 



(l-§(flij) logP(Bi = Q 



S(fli) log P [Bi = l 



= Z & 



(ii) 
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where Z B = Ym=i 1°§ V- ~ PbW) 1S a constant and 
S B (i) = log (pg(i)/(l — p^(i))) is the logit function of 
P B W- 

The discrete BFs used in this work arise from two 
sources: background subtraction and shape priors. To 
compute a discrete BF Bf using the Background Sub- 
traction technique [22] . recall the probability densities 
p Q . (/(A)k(A) = 0) and m (/(^)k(^) = 1) defined 
in Section|4]to model the probability of observing a fea- 
ture f(fii) at a given pixel fii, depending on the pixel's 
state q{fii). The success rates of the discrete BF Bf are 
thus defined as 



Pb,W - 



pa i (f(n i )\q(O i ) = l) 



pn t (/(«i)|g(/2i) = 0) +pn 4 (/(/2 i )|«(A) = 1)' 

(12) 



To compute a discrete BF B s associated with a efe- 
crete shape prior, we can estimate the success rates 
p B (i) of B s from a collection of N discrete shapes, 

£ = | Si, . . . , Sjv|, assumed to be aligned in the set 
fi. These discrete shapes can be acquired by different 
means, e.g., using a 2D or 3D scanner, for d = 2 or 
d = 3, respectively. The success rate p B (?) of a particu- 
lar Bernoulli variable Bi (i — 1, . . . , n) is thus estimated 
from |5i(J7i), . . . , Sjv(^i)j using the standard for- 
mula for the estimation of Bernoulli distributions 1 16 , 



1 N 



(13) 



Discrete shapes, as in Definition [2] have two lim- 
itations that must be addressed to enable subsequent 



developments. First, the log-probability in ( 11 1 depends 



(implicitly) on the unit size of the partition (which is 
related to the image resolution), preventing the com- 
parison of log-probabilities of images acquired at dif- 
ferent resolutions (this will be further explained after 
Definition [6]) . Second, it was assumed in (11) that the 
Bernoulli variables Bi and the pixels fii were perfectly 
aligned. However, this assumption might be violated 
if a transformation (e.g., a rotation) is applied to the 
shape. To overcome these limitations, and also to fa- 
cilitate the proofs that will follow, we introduce next 
the second shape representation, that of a continuous 
shape. 



5.2 Continuous shapes 

Definition 4 (Continuous shapes) Given a set fi C 
R d , we define a continuous shape S to be a function 



S : Q — > {0, 1} (Fig. |5p,). We will often abuse notation 
and refer to the set S = {x £ fi : S(x) — 1} also as 
the shape. To avoid pathological cases, we will require 
the set S to satisfy two regularity conditions: 1) to be 
open (in the usual topology in K d [T3]) and 2) to have 
a boundary (as defined in |13| ) of measure zero. 

Given a discrete shape S defined on a partition 
n{Q) — {fii, . . . , fi n }, the continuous shape S(x) = 
S(fii) Vx 6 Qi, is referred to as the continuous shape 
produced by the discrete shape S, and is denoted as 
S ~ S or S ~ S. Intuitively, S extends S from every 
element of n(fl) to every point x e fl. 

We would like now to extend the definition of the 
log-probability of a discrete shape (in Definition |3| to 
include continuous shapes. Toward this end we first in- 
troduce continuous BFs, which play in the continuous 
case the role that discrete BFs play in the discrete case. 

Definition 5 (Continuous Bernoulli Fields) Given 
a set fi C M. d , a continuous Bernoulli field (or simply 
a BF) is the construction that associates a Bernoulli 
random variable B x to every point x € fl. The success 
rate for each variable in the field is given by the func- 
tion Ps(x) = P(B X = 1). The corresponding logit func- 
tion <5_b(x) = log ( i-p^x) ) ana - constant term Zb — 
J n log (1 — pg(x.))dx are as in Definition [3] 

We will only consider in this work functions ps(x) 
such that \Zb\ < 00 and #b(x) is a measurable func- 
tion 03]. Furthermore, since e < ps(x) < 1 — e Vx 6 J?, 
5 B (x) £ (S max , S max ) Vx e ft, with S max = log 
Note that a BF is not associated to a continuous prob- 
ability density on fl (e.g., it almost never holds that 
J^Pb(x) <ix = 1), but rather to a collection of discrete 
probability distributions, one for each point in fl (thus, 
it always holds that P (B x = 0) + P {B x = 1) = 1 Vx e 
Q). 

Due to the finite resolution of cameras and scanners, 
continuous BFs cannot be directly obtained as discrete 
BFs were obtained in Definition [3] In contrast, con- 
tinuous BFs are obtained indirectly from discrete BFs 
(which are possibly obtained by one of the methods de- 
scribed in Definition [3J. Let B be a discrete BF defined 
on a partition n(Q) = . . . , ]?„}. Then, for each 
partition element fJi, and for each point x G J?j, the 
success rate of the Bernoulli variable B x is defined as 
Pb(x) = p B (i)- The BF B produced in this fashion will 
be referred to as the BF produced by the discrete BF B. 
Intuitively, p B extends p^ from every element of 77(i7) 
to every point x e fi. Note that this definition is anal- 
ogous to the definition of a continuous shape produced 
from a discrete shape in Definition [4] 
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Let fl' C M. d be a set referred to as the canonical 
set, let fl C K d be a second set referred to as the world 
set, and let T :]?'—> J? be a bijective transformation 
between these sets. Given a BF B in fl' with success 
rates ps(x), the transformed BF £>r in ^ is defined as 
-Bt = Bo T^ 1 with success rates ps T (x) = (T _1 x) . 

Definition 6 (Log-probability of a continuous sha- 
pe) Let B be a BF in fl with success rates given by 
the function pg(x), let S be a continuous shape also 
in fl, and let u > be a scalar called the equivalent 
unit size. We define the log-probability that a shape S is 
produced by a BF B, by extension of the log-probability 



of discrete shapes in (111, as 



logP(B = S) 



1 



Zb + S(x)Sb(x) dx 
J n 



where Zb and 8b are respectively the constant term 
and the logit function of B. 

Several things are worth noting in this definition. 
First, note that if there is a uniform partition TI(fl) = 
. . . , fl n } with \ fli\ = u Q Vi, and if the continuous 
shape S and the BF B are respectively produced by 
the discrete shape S and the discrete Bernoulli field B 
defined on 11(0), then \ogP{B = S) = logP (b = S 



For this reason we said that the definition in ( 14 ) ex- 
tends the definition in (11). However, keep in mind 



that in the case of a continuous shape ( 14 1 is not a 
log-probability in the traditional sense, but rather it ex- 
tends the definition to cases in which S'(x) is not pro- 
duced from a discrete shape and 5b ( x ) is not piecewise 
constant in a partition of fl. 



Second, note that while (14 1 provides the "log-pro 



bability" density that a given continuous shape is pro- 
duced by a BF, sampling from a BF is not guaranteed 
to produce a continuous shape (because the resulting 
set might not satisfy the regularity conditions in Def- 
inition [4]). Nevertheless, this is not an obstacle since 
in this work we are only interested in computing log- 
probabilities of continuous shapes that are given, not 
on sampling from BFs. 



Third, note in ( 14 1 that the log-probability of a con- 



tinuous shape is the product of two factors: 1) the in- 
verse of the unit size, which only depends on the par- 
tition (but not on the shape); and 2) a term (in brack- 
ets) that does not depend on the partition. In the case 
of continuous shapes, u in the first factor is not the 
unit size of the partition (there is no partition defined 
in this case) but rather a scalar defining the unit size 
of an equivalent partition in which the range of log- 
probability values obtained would be comparable. The 
second factor is the sum of a constant term that only 



depends on the BF B, and a second term that also de- 
pends on the continuous shape S. 

Fourth, the continuous shape representation in Def- 
inition[4]overcomes the limitations of the discrete repre- 
sentation pointed out above. More specifically, by con- 



sidering continuous shapes, (14) can be computed even 



if a discrete shape and a discrete BF are defined on par- 
titions that are not aligned, allowing us greater freedom 
in the choice on the transformations (T) that can be ap- 
plied to the BF. Furthermore, the role of the partition 
is "decoupled" from the role of the BF and the shape, 



allowing us to compute ( 14 ) independently of the reso- 
lution of the partitions. 



(14) 5.3 Semi-discrete shapes 



As mentioned at the beginning of this section, discrete 
shapes will be used to obtain a lower bound for the log- 
probability of a continuous shape. Unfortunately, upper 
bounds for the log-probability derived using discrete 
shapes are not very tight. For this reason, to obtain 
upper bounds for this log-probability, we need to intro- 
duce the third shape representation, that of semidis- 
crete shapes. 

Definition 7 (Semidiscrete shapes) Given a par- 
tition TI(fl) = {Hi, ... , Q n } of the set Q C W l , the 
semidiscrete shape S is defined as the function S : 
TI(fl) — > [0, \fl\], that associates to each element fli 
in the partition a real number in the interval [0, \fli\], 
i.e., S(f2i)e [0,|fli|] (Fig. §:). 

Given a continuous shape S in fl, we say that this 
shape produces the semidiscrete shape S, denoted as 
S ~ S or S ~ S, if S(fli) = \Snf2-\ for i = 1, . . . , n. 
An intuition that will be useful later to understand the 
derivation of the upper bounds is that a semidiscrete 
shape produced from a continuous shape "remembers" 
the measure of the continuous shape inside each clement 
of the partition, but "forgets" where exactly the shape 
is located inside the element. 

Given two continuous shapes in fl, Si and S2, that 
produce the the same semidiscrete shape S in the par- 
tition II(fl), we say the these continuous shapes are 
related (denoted as Si ~ 82). The nature of this rela- 
tionship is explored in the next proposition. 

Proposition 1 (Equivalent classes of continuous 
shapes) The relationship "~ " defined above is an equiv- 
alence relation. 

Proof: The proof of this proposition is trivial from Def- 
inition^ □ 
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We will say that these continuous shapes are equivalent 
in the partition 77(12), and by extension, we will also 
say that they are equivalent to S (i.e., Si ~ S). 

Proposition 2 (Relationships between shape rep- 
resentations) Let 77(i?) = {J?i, . . . , Sl n } be an arbi- 
trary partition of a set SI, and let S(J7) and S(i7) be 
the sets of all discrete and semidiscrete shapes, respec- 
tively, defined on LJ (SI). Let S be the set of all contin- 
uous shapes in SI. Then, 



{5:5~5,5e§(JJ)}cS, and, 
{S:S~S,SeS(II)\=S. 



(15) 
(16) 



Proof: The proof of this proposition is trivial from def- 
initions^^ and^ □ 

5.4 LCDFs and summaries 

So far we have introduced three different shape repre- 
sentations and established relationships among them. 
In this section we introduce the concepts of logit cumu- 
lative distribution functions (LCDFs) and summaries. 
These concepts will be necessary to use discrete and 
semidiscrete shapes to bound the log-probability of con- 
tinuous shapes, and hence, to bound the evidence. 

Intuitively, a LCDF "condenses" the "information" 
of a BF in a partition element into a monotonous func- 
tion. A summary then further "condenses" this "infor- 
mation" into a single vector of fixed length. Impor- 
tantly, the summary of a BF in a partition element 
can be used to bound the evidence L(H) and can be 
computed in constant time, regardless of the number of 
pixels in the element. 

After formally defining LCDFs and summaries be- 
low, we will prove in this Section some of the properties 
that will be needed in Section [6] to compute lower and 
upper bounds for 7(77). In the remainder of this sec- 
tion, unless stated otherwise, all partitions, shapes and 
BFs are defined on a set SI C R d . 

Definition 8 (Logit cumulative distribution func- 
tion) Given the logit function 8b of a BF B and a 
partition LJ, the logit cumulative distribution function 
(or LCDF) of the BF B in LI is the collection of func- 
tions Db — {Db.u} uen , where each function Db,u ■ 



J max 7 ^max\ 



[0, |w|] is defined as 



D BlU) (6) 4 |{x G w : M*) < <*}l (w G 77). 



(17) 



It must be noted that this definition is consistent, 
since from Definition [5j the logit function is measur- 
able. The LCDF is named by analogy to the proba- 
bility cumulative distribution function, but must not 




8,. 



+ 



^max 

Fig. 6 Plot of the LCDF a - D B ,u(S) (left) and its inverse 
5 = Dg u (a) (right). The shaded area under the curve on the 
right is the maximal value of the integral on the rhs of \2Q\ 
for any BF B ~ Db,u ( see Lemma 

be confused with it. Informally, a LCDF "condenses" 
the information of the BF by "remembering" the val- 
ues taken by the logit function inside each partition 
element, but "forgetting" where those values are inside 
the element. This relationship between BFs and their 
LCDFs is analogous to the relationship between con- 
tinuous and semidiscrete shapes. 



Equation (17) defines a non-decreasing and possi- 



bly discontinuous function. To see that this function 
is non-decreasing, note that the set \i — { x G w : 
5b (x) < Si} is included in the set \2 — {x G cj : 
8b (x) < ($2} if Si < 62- Moreover, this function is not 
necessarily strictly increasing because it is possible to 
have 7?b jW ((5i) = Db,uj{S2) with <5i < 82 if the measure 
of the set {x G u> : Si < <5_b(x) < 82} is zero. To see 



that the function defined in (17 1 can be discontinuous, 



note that this function will have a discontinuity when- 
ever the function 8b (x) is constant on a set of measure 
greater than zero. 

Later in this section we will use the inverse of a 
LCDF, Dg \j(a). If Db,u>(8) were strictly increasing and 
continuous, we could simply define Dg 1 UJ (a) as the unique 
real number 5 G [— 8 max , 8 ma x] such that Db.u{8) = a. 
For general LCDFs, however, this definition does not 
produce a value for every a € [0, \u\]. Instead we use 
the following definition. 

Definition 9 (Inverse LCDF) The inverse LCDF 
Dg^a) is defined as (see Fig. m 



D^a) ± w£ {6 : DbM > "} > 



(18) 



To avoid pathological cases, we will only consider in this 
work LCDFs whose inverse is continuous almost every- 
where (note that this imposes an additional restriction 
on the logit function). 



14 



Definition 10 (X-axis crossing point) We define 
the x-axis crossing point, A B .ui as 



D B , U (Q-) + D BjU1 (0+) 



(19) 



This quantity has the property that D B l u {a) > if 
a > A B , U , and D B ^(a) < if a < A ByU1 . 

Definition 11 (Summaries) Given a partition 77, a 
summary of a LCDF D B = {D BlUi } wen m ^ ne Peti- 
tion is a functional that assigns to each function D B u 
(uj e 77) in the LCDF a vector Y B ,u € R dY ■ The name 
"summary" is motivated by the fact that the "infinite 
dimensional" distribution is "summarized" by just dy 
real numbers. 

Two types of summaries are used in this article: 
m-summaries and mean-summaries. Given m > 0, an 
m-summary assigns to each function D B ^ in the LCDF 



the (2rn+l)-dimensional vector Y B ^ = Y B 
of equispaced samples of the LCDF, i.e., 

jSmc 



Y, 



B M 



y B ^ = D 



B ,U 



-m, . 



,mi 



(20) 



Note that since the LCDF is known to be a non-decrea- 
sing function, the information in the m-summary can be 
used to bound the inverse LCDF (Fig. [7}. Specifically, 

we know that D^Ja) < ^(j + 1) for a G [y^ u , 
yL + ] , which can be written as 



D B ^(a)<D Y ^Ja) 

A S, 



J(o), Va G [0, |w[ 



(21) 



with 



J (a) = minjj :Y Buj > 



} 



(22) 



These bounds will be used in turn to compute an upper 
bound for L(H). 

The second type of summaries, referred to as mean- 
summaries, will be used to compute a lower bound 
of L(H). The mean-summary assigns to each function 
D B u the scalar 



Yb,u = / D B ^( a ) da= 5 B (x) dx. 

JO Ju 



(23) 



The last equality follows by setting S(u>) = \oj\ in (29 1 
and is proved later in Lemma [T] Note that this equal- 
ity provides the means to compute the mean-summary 
directly from the logit function, without having to com- 
pute the LCDF. 



Si 
= S 

S-l 



V 1 

rsi 
1 


\ 




V 

1 S?' 




































I 1 1 >a 























Fig. 7 Lower (in orange) and upper (in green) bounds for the 
inverse LCDF S = Dg (a) (in black) computed from the m- 
summaries with m — 2. The j-th component of the summary, 
Y B , is represented by a dotted vertical line at the horizontal 



position a 



Y 3 

B, 



(or where 7? B 1 „(a) = 8j = jS ma x/m). 



One of the main properties of summaries is that, for 
certain kinds of sets, they can be computed in constant 
time regardless of the number of elements (e.g., pixels) 
in these sets. Next we show how to compute the mean- 
summary Y Bj & and the m-summaries Y B ^> of a BF B 
for the set <P C (2 (defined below). For simplicity we 
assume that Q C K 2 , but the results presented here im- 
mediately generalize to higher dimensions. We also as- 
sume that 77(12) = { i , L?i.2, ■ ■ • , , f2 n ,n} is a uniform 
partition of fl organized in rows and columns, where 
each partition element fi^j (in the i-th row and the j- 
th column) is a square of area \ = uq. We assume 
that B, defined by its logit function <5b(x) (x € Q), 
was obtained from a discrete shape prior B in 77(L?) 
(as described in Definition [H| , and therefore 5 B (x) = 
S B (i,j) Vx G L7.IJ. And finally, we assume that <P is an 
axis-aligned rectangular region containing only whole 
pixels (i.e., not parts of pixels). That is, 



<7>^ 



U Q ^r 



(24) 



i L <i<iu 
3l<3<3u 



In order to compute the mean-summary Y B &, note 



that from ( 23 ) 



Y, 



B .<i> 



E 



<5b(x) dx 



ih<i<iu 
3l<3<3u 



U o E 5 fi- 



i L <i<i v 
3l<3<3u 



The sum on the rhs of ( 25 ) can be computed in con- 
stant time by relying on integral images |41j . an image 
representation precisely proposed to compute sums in 
rectangular domains in constant time. To accomplish 
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this, integral images precompute a matrix where each 
pixel stores the cumulative sum of the values in pixels 



with lower indices. The sum in ( 25 ) is then computed as 



the sum of four of these precomputed cumulative sums. 
The formula to compute the m-summary Yg # is 



similarly derived. From (20), and since 5b is constant 
inside each partition clement, it holds for k = — m, . . . , m 
that 



I B.<P — 



x e $ : <5 B (x) < 



i L <i< iu,jL <j< ju,S B (i,j) < 



kS n 



Let us now define the matrices Ik (k - 

m ax 

jra 

k[h3) ~ 1 0, otherwise. 



-m, . 



Using this definition, ( 26 ) can be rewritten as 



(26) 
. , to) as 

(27) 
(28) 



i L <i<iu 

jL<j<ju 

which as before can be computed in O(l) using integral 
images. 

Before deriving formulas to bound £(77), we need 
the following two results. 

Proposition 3 (Equivalent classes of BFs) Let LJ 

be a partition, let B\ and B 2 be two BFs, and consider 
the following binary relations: 1) B\ and B2 are re- 
lated ( denoted as B\ ~ D B 2 ) if they produce the same 
LCDF D = {D UJ } ujen ; and 2) B\ and B 2 are related 
( denoted as B x ~ Y B 2 ) if they produce the same sum- 
mary Y = {Y u }^ en . Then, "~d" and "^ Y " are equiv- 
alence relations. 



Proof: Immediate from definitions^ and 11 



In the first case (~z>) we will say that these BFs 
are equivalent in the partition 77 with respect to dis- 
tributions. Abusing the notation, we will also say that 
they are equivalent to (or compatible with) the LCDF 
(i.e., Bi ~ D). Similarly, in the second case (~y) we 
will say that these BFs are equivalent in the partition 
7J with respect to summaries. Abusing the notation, we 
will also say that they are equivalent to (or compatible 
with) the summary (i.e., Bi ~ Y). Note that if two BFs 
are equivalent with respect to a LCDF, they are also 
equivalent with respect to any summary of the LCDF. 
The reverse, however, is not necessarily true. 



Lemma 1 (Properties of LCDFs and m-Summa- 
ries) Let LI be an arbitrary partition, and let S and 
D = {Dui} u cn oe respectively a semidiscrete shape and 
a LCDF in this partition. Then: 

1. Given a BF B such that B ~ D, it holds that 
(Fig. § 

sup f 5 B (x)S(x.) dx = V t ' Dj 1 (a) da. (29) 



2. Similarly, given a continuous shape S , such that 
S ~ S, it holds that 



sup / Mx)S(x) dx= f" D ^( a ) 



da. 



(30) 



3. Moreover, for any a £ [0, \u>\], the integrals on 
:an be bounded as 

J(a) - a) + 



the rhs of ( 29 ) and ( 30 ) can be bounded as 

M x 
DzHa) da<°-^_ 



III — 1 



3=A<x) 



(31) 



where J (a) is as defined in ( 22 ) 



4- The rhs of (29) and (30) are maximum when 



S(uj) = — A u , thus 



,\u\ n\u\ 

supE / D- 1 (a)da=J2 D Z\a) da 

S uen J\u\-S( u ) uen jA - 



(32) 



Proof: Due to space limitations this proof was included 
in the supplementary material. □ 

This concludes the presentation of the general math- 
ematical framework to compute and bound probabili- 
ties of shapes. This framework is used in the next sec- 
tion to bound the evidence for the problem defined in 
Section |4j and also in [32] to bound the evidence for a 
more complex problem. 



6 Bounding mechanism 

We are now ready to develop the BM for the problem 
defined in Section [3J This BM is the second part of the 
H&B algorithm proposed to solve this problem. To de- 
velop the BM we proceed in three steps: first we derive 
an expression for the evidence L(H) that the FoAM 



will maximize (Section 6.1); second we show how to 



compute lower and upper bounds for the evidence for 
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a given partition (Section 6.2 1; and third, we show how 



to construct, incrementally, a sequence of increasingly 
finer partitions that will result in increasingly tighter 



bounds (Section 6.3 1. 



6.1 Definition of the Evidence L(H) 

In order to define the evidence L{H) for the problem, 
we re-label the pixels of the image in ^ (denoted pre- 
viously by x) as the elements of a uniform partition 
77, and separate the factors according to its state q(oj). 
Hence ^ is equal to 

P(H) JJ P u (/M|g( W ))P(g( W )|ff) = 

P(H) J] [^(/MkH=0)P(gH=0|i/)] 1 ^ M x 

x [P u (/( W )|g( W ) = 1)P(?H = 1|P)] 9M . 

(33) 

Defining a BF Bf based on the features observed in the 
input image, with success rates given by 



PBf H 



P u (f(u)\q(u) = 1) 



P, (/(w)|g(w) = 0) + P w - 1) ' 



(34) 



assuming that all hypotheses are equally likely, and di- 
viding (|33[) by the constant (and known) term 



p(p) II (*U/MlsM = o) + p w (/(w)|?H = i)), 

(35) 

we obtain an expression equivalent to ([7]), 



n 



(l-p Bf (u))(l-p BH (u)) 
, <?(") 

p B/ (w)p Sff (w) 



(36) 



(recall from Section [4] that B B is a BF with success 
rates p BfI — P{q{u)) = 1|77))- This expression can be 
further simplified by taking logarithms and using the 
variables introduced in Definition [3] to yield 

Z B} +Z Bh + J2 «M ( 6 B f M + 5b h (w)) ■ (37) 

Using the extension to the continuous domain ex- 
plained in Definition [6j and substituting (37) into 
(JqJ) can be rewritten as 

7'(77) =sup — 



Z f 



q(x) (S Bf (x) + S Bh (x)) dx 



(38) 



Now, since Z Bf and u c are constant for every hypothesis 
and every continuous shape q, maximizing this expres- 
sion is equivalent to maximizing 



7(77) = Z Bh + sup / q(x) (S Bf (x) + 5 Bh (x)) dx. 
q Jn 



(39) 



This is the final expression for the evidence. Due to 
the very large number of pixels in a typical image pro- 
duced by a modern camera, computing 7(77) directly 



as the integral in (39 1 would be prohibitively expensive. 



For this reason, the next step is to derive bounds for 



(39) that are cheaper to compute than (39) itself, and 



are sufficient to discard most hypotheses. These bounds 
are derived in the next section. 



6.2 Derivation of the bounds 

In the following two theorems we derive bounds for the 
evidence of a hypothesis that can be computed from 



summaries of the BFs Bf and B B (in (39)), instead of 
computing them from the BFs directly. Because sum- 
maries can be computed in 0(1) for each element in a 
partition, bounds for a given partition can be computed 
in 0(n) (where n is the number of elements in the par- 
tition) , regardless of the actual number of pixels in the 
image. 

Theorem 1 (Lower bound for L(H)) Let LT be a 

partition, and let Yt — { Yt w > and Y B — \ Ya u \ 

be the mean- summaries of two unknown BFs in LT. 
Then, for any Bf ~ Yt and any B B ~ Y B , it holds 
that 7(77) > L n (H), where 



L n (H)^Z BH + J2^(H), 

£^{H) = (Y fiU + Y H , U ) &(w). 

and <f* is a discrete shape in LT defined as 



q* M = 



1, if(Y ftU + Y H>u ) 
0, otherwise. 



(40) 
(41) 

(42) 



Proof: Due to space limitations this proof was included 
in the supplementary material. □ 

Theorem 2 (Upper bound for L(H)) Let LI be a 
partition, and let Yt — { Yt w > and Y B = \ Ya u \ 
be the m- summaries of two unknown BFs in LL. Let 
5/©H,w € LT) be a vector of length 4m + 2 obtained 
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by sorting the values in Yf u and Yh.u> (in ascending 
order), keeping repeated values, i.e., 



Y, 



f®H,u 



V 1 



y4m+2 
' 1 f®H,u 



= be 



rtAscending (ff^ U Y H ,uj ■ (43) 



Then, for any Bf ~ Yf and any Bh ~ Yh, it holds 
that L(H) < L n (H), where 



L n {H) ^Z Bh + J2 ^(H), and 
u>en 

x 4m+l 



(44) 



Y 3 



(45) 



7£ also follows that the continuous shape that maximizes 



( 39 ) is equivalent to a semidiscrete shape in II that 



satisfies 
q* M G 



2m+l 



M - Y, 



f®H,u 



\fw e n. (46) 



Proof: Due to space limitations this proof was included 
in the supplementary material. □ 

Theorems 1 and 2 presented formulas to compute 
lower and upper bounds for L(H), respectively, for a 
given partition 77. Importantly, these theorems also in- 
clude formulas to compute a discrete shape q* and a 
semidiscrete shape q* that approximate (in the parti- 



tion 77) the continuous shape q that solves ( 39 ) . In the 
next section we show how to reuse the computation 
spent to compute the bounds for a partition 77/j, to 
compute the bounds for a finer partition n k+ i. 



6.3 Incremental refinement of bounds 

Given a partition 77^ containing hk elements, it can 
be seen in (|40| and Jiil that the bounds for the evi- 



dence corresponding to this partition can be computed 
in 0(h k ). In Section |3j however, we requested that the 
BM be able to compute these bounds in 0(1). In order 
to compute a sequence of progressively tighter bounds 
for a hypothesis 77, where each bound is computed in 
O(l), we inductively construct a sequence of progres- 
sively finer partitions of J? for the hypothesis. 

Let us denote by 77^(77) = {Oh,i> ■ ■ ■ , ^H,h k } the 
fc-th partition in the sequence corresponding to 77. Each 
sequence is defined inductively by 

77i(77) = {i?}, and (47) 
7T fc+1 (77) 4 [7T fe (77) \ u h ] U 7r(w fc ), (k > 0), (48) 



where u>k G 77^(77) and 7r(wfc) is a partition of w^. For 
each partition 77^(77) in the sequence, lower (L_ k {H)) 
and upper (7^(77)) bounds for the evidence 7v(77) could 



be computed in O(k) using (40) and (44), respectively. 
However, these bounds can be computed more efficiently 
by exploiting the form of (Hsj), as 



L k+l {H) = L k {H)-C Wk {H)+ (49) 

cue7r(w fc ) 

(A similar expression for the upper bound Lk+i(H) can 
be derived.) If the partition of Wfc, n(uJk), is chosen to 
always contain a fixed number of sets (e.g., |7r(a>fc)| = 
4 Vfc > 1), then it can be seen in (49) that 0(1) evalu- 



ations of (41) are required to compute L k+1 (H). 

While any choice of ujk from 77^ (H) in ( 48 ) would 



result in a new partition 77fe + i(77) that is finer than 
77fc(7J), it is natural to choose uik to be the set in 77^(77) 
with the greatest local margin (£ Uk (H) ~£w k (^0) since 
this is the set responsible for the largest contribution to 
the total margin of the hypothesis (Lk(H) — L k (77)). In 
order to efficiently find the set u>k with the greatest local 
margin, we store the elements of a partition in a priority 
queue, using their local margin as the priority. Hence 
to compute 77fe + i(77) from 77^(77) (in ( |48| ) we need 
to extract the element u>k of the queue with the largest 
local margin, and then insert each element in 7r(wfc) into 
the queue. Taken together these steps have, depending 
on the implementation of the queue, complexity of at 
least O (log hk) (where hk is the number of elements 
in the partition) |28j . In our case, however, it is not 
essential to process the elements in strictly descending 
margin order. Any element with a margin close enough 
to the maximum margin would produce similar results. 
Moreover, in our case we know that the margins belong 
to the interval (0, Cq(H) — C n (H)\ and that they tend 
to decrease with time. 

Based on these considerations we propose a queue 
implementation based on the untidy priority queue of 
Yatziv et al. [44], in which the operations GetMax and 
Insert both have complexity 0(1). This implementa- 
tion consists of an array of buckets (i.e., singly-linked 
lists), where each bucket contains the elements whose 
margin is in an interval Ij. Specifically, suppose that 
the minimum and maximum margin of any element 
are known to be M_ and Ai, respectively, and that 
p > 1 is a constant (we chose p — 1.2). The inter- 
vals are then defined to be Ij = [Mp 3 ^ 1 . Mp>) (j = 
1, . . . , [logp M./JA] , where ["•] is the ceiling function). 
To speed up the GetMax operation, a variable j max 
keeps the index of the non-empty bucket containing the 
element with the largest margin. In the Insert opera- 
tion, we simply compute the index j of the correspond- 
ing bucket, insert the element in this bucket, and up- 
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date jmax if j > jmax- In the GetMax operation, we re- 
turn any element from the jmax-^i bucket, and update 
jmax- Note that the margin of the returned element is 
not necessarily the maximum margin in the queue, but 
it is at least 1 / p times this value. Since both operations 
(insert and GetMax) can be carried out in 0(1), we 



have proved that (49) can also be computed in O(l). 



Moreover, since the bounds in (41) and (45) arc 



tighter if the regions involved are close to uniform (be- 
cause in this case, given the summary, there is no uncer- 
tainty regarding the value of any point in the region), 
this choice of 0Jk automatically drives the algorithm to 
focus on the edges of the image and the prior, avoid- 
ing the need to subdivide and work on large uniform 
regions of the image or the prior. 

This concludes the derivation of the bounds to be 
used to solve our problem of interest. In the next section 
we show results obtained using these bounds integrated 
with the FoAM described in Section [3l 



7 Experimental results 

In this section we apply the framework described in 
previous sections to the problem of simultaneously es- 
timating the class, pose, and a denoised version (a seg- 
mentation) of a shape in an image. We start by ana- 
lyzing the characteristics of the proposed algorithm on 



synthetic experiments (Section 7.1), and then present 

These experi- 



7.2) 



experiments on real data (Section 
ments were designed to test and illustrate the proposed 
theory only. Achieving state-of-the-art results for each 
of the specific sub-problems would require further ex- 
tensions of this theory. 



7.1 Synthetic experiments 

In this section we present a series of synthetic experi- 
ments to expose the characteristics of the proposed ap- 
proach. 

Experiment 1. We start with a simple experiment where 
both the input image (Fig. [8^,) and the shape prior are 
constructed from a single shape (the letter 'A'). Since 
we consider a single shape prior, we do not need to es- 
timate the class of the shape in this case, only its pose. 
In this situation the success rates ps f of the BF corre- 
sponding to this image (Fig. ^p) , and the success rates 
Pb k of the BF corresponding to the shape prior, are 
related by a translation t (i.e., Pb, (x) = pb k (x — t)). 
This translation is the pose that we want to estimate. 
In order to estimate it, we define four hypotheses and 
use the proposed approach to select the hypothesis H 



A 




(a) 

Fig. 8 (a) The input image considered in Experiment 1 
and (b) its corresponding BF with success rates Pb,- (Inset) 
Zoom-in on the edge of the 'A'. The rectangles in (b) indi- 
cate the supports corresponding to each of the 4 hypotheses 
defined in this experiment. The same colors are used for the 
same hypotheses in the following two figures. Throughout this 
article, use the colorbar on the right of this figure to interpret 
the colors of the BFs presented. 

that maximizes the evidence L(H). Each hypothesis is 
obtained for a different translation (Fig. ^p), but for 
the same shape prior. 

As described in Section [3j at the beginning of the 
algorithm the bounds of all hypotheses are initialized, 
and then during each iteration of the algorithm one hy- 
pothesis is selected, and its bounds are refined (Fig. 
[9]). It can be seen in Fig. [9] that the FoAM allocates 
more computational cycles to refine the bounds of the 
best hypothesis (in red), and less cycles to the other 
hypotheses. In particular, two hypotheses (in cyan and 
blue) are discarded after spending just one cycle (the 
initialization cycle) on each of them. Consequently, it 
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Fig. 9 Progressive refinement of the evidence bounds ob- 
tained using a H&B algorithm, for the four hypotheses de- 
fined in Fig. [8] The bounds of each hypothesis are repre- 
sented by the two lines of the same color (the lower bound 
for the hypothesis in cyan, however, is occluded by the other 
lower bounds). During the initialization stage [cycle < 0), the 
bounds of all hypotheses are initialized. Then, in the refine- 
ment stage (cycle > 1), one hypothesis is selected in each cy- 
cle and its bounds are refined (this hypothesis is indicated by 
the marker 'o'). Hypotheses 4 (cyan), 3 (blue), and 2 (green) 
are discarded after the 2nd, 13th, and 19th refinement cy- 
cles, respectively, proving that Hypothesis 1 (red) is optimal. 
Note that it was not necessary to compute the evidence ex- 
actly to select the best hypothesis: The bounds are sufficient 
and much cheaper to compute. 
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Fig. 10 Final partitions obtained for the four hypotheses 
in Experiment 1. The color of each element of the partition 
(i.e., rectangle) indicates the margin of the element (use the 
colorbar on the right to interpret these colors). Note that 
higher margins are obtained for the elements around the edges 
of the image or the prior, and thus these areas are given 
priority during the refinements. The edges of the letter 'A' 
were included for reference only. 

can be seen in Fig. [10] that the final partition for the 
red hypothesis is finer than those for the othe hypothe- 
ses. It can also be seen in the figure that the partitions 
are preferentially refined around the edges of the im- 



age or the prior (remember from ( 39 1 that image and 
prior play a symmetric role), because the partition ele- 
ments around these areas have greater margins. In other 
words, the FoAM algorithm is "paying attention" to the 
edges, a sensible thing to do. Furthermore, this behav- 
ior was not intentionally "coded" into the algorithm, 
but rather it emerged as the BM greedily minimizes 
the margin. 

To select the best hypothesis in this experiment, the 
functions to compute the lower and upper bounds (i.e., 



those that implement (41)- (42 1 and ( 45 )-( 46 1) were ca- 
lled a total of 88 times each. In other words, 22 pairs of 
bounds were computed, on average, for each hypothe- 
sis. In contrast, if L(H) were to be computed exactly, 
16,384 pixels would have to be inspected for each hy- 
pothesis (because all the priors used in this section were 
of size 128 x 128). While inspecting one pixel is signifi- 
cantly cheaper than computing one pair of bounds, for 
images of "sufficient" resolution the proposed approach 
is more efficient than naively inspecting every pixel. 
Since the relative cost of evaluating one pair of bounds 
(relative to the cost of inspecting one pixel) depends on 
the implementation, and since at this point an efficient 
implementation of the algorithm is not available, we use 
the average number of bound pairs evaluated per hypoth- 
esis (referred as r) as a measure of performance (i.e., 
t = Pairs of bounds computed/Number of Hypotheses). 

Moreover, for images of sufficient resolution, not 
only is the amount of computation required by the pro- 
posed algorithm less than that required by the naive 
approach, it is also independent of the resolution of 
these images. For example, if in the previous experi- 
ment the resolution of the input image and the prior 
were doubled, the number of pixels to be processed by 
the naive approach would increase four times, while the 
number of bound evaluations would remain the same. 



In other words, the amount of computation needed to 
solve a particular problem using the proposed approach 
only depends on the problem, not on the resolution of 
the input image. 

Experiment 2. The next experiment is identical to the 
previous one, except that one hypothesis is defined for 
every possible integer translation that yields a hypoth- 
esis whose support is contained within the input im- 
age. This results in a total of 148,225 hypotheses. In 
this case, the set A of active hypotheses when termina- 
tion conditions were reached contained 3 hypotheses. 
We refer to this set as the set of solutions, and to each 
hypothesis in this set as a solution. Note that having 
reached termination conditions with a set of solutions 
having more than one hypothesis (i.e., solution) implies 
that all the hypotheses in this set have been completely 
refined (i.e., either pB f or pb h are uniform in all their 
partition elements). 

To characterize the set of solutions A, we define the 
translation bias, /it, and the translation standard devi- 
ation, <7t, as 
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(50) 



(51) 



respectively, where tj is the translation corresponding 
to the i-th hypothesis in the set A and is the true 
translation. In this particular experiment we obtained 
/it = and at = 0.82, and the set A consisted on 
the true hypothesis and the two hypotheses that are 
one pixel translated to the left and right. These 3 hy- 
potheses are indistinguishable under the conditions of 
the experiment. There are two facts contributing to the 
uncertainty that makes these hypotheses indistinguish- 
able: 1) the fact that the edges of the shape in the image 
and the prior are not sharp (i.e., not having probabil- 
ities of 0/1, see inset in Fig. ^p); and 2) the fact that 
m < oo in the m-summaries and hence some "informa- 
tion" of the LCDF is "lost" in the summary, making 
the bounds looser. 

Figure [TT] shows the evolution of the set A of ac- 
tive hypotheses as the bounds get refined. Observe in 
this figure that during the refinement stage of the algo- 
rithm (cycles > in the figure), the number of active 
hypotheses (|A|), the bias /it, and the standard devia- 
tion <7t sharply decrease. It is interesting to note that 
during the first half of the initialization stage, because 
hypotheses are not discarded symmetrically around the 
true hypothesis, the bias increases. 
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Fig. 11 (Left) Evolution of the number of live hypotheses 
(|A|) as a function of time. (Right) Evolution of the transla- 
tion bias /j,t (blue) and standard deviation at (green) of the 
set of active hypotheses A. 
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Fig. 12 Percentage 
of the hypotheses that 
were refined a certain 
number of times. 
Most hypotheses 
(95.5%) were refined 
only once, or not 
refined at all. 



Figure [12] shows the percentage of hypotheses that 
were refined or 1 times, between 2 and 9 times, be- 
tween 10 and 99 times, and between 100 and 999 times. 
This figure indicates that, as desired, very little com- 
putation is spent on most hypotheses, and most com- 
putation is spent on very few hypotheses. Concretely, 
the figure shows that for 95.5% of the hypotheses, ei- 
ther the initialization cycle is enough to discard the 
hypothesis (i.e., only 1 pair of bounds needs to be com- 
puted), or an additional refinement cycle is necessary 
(and hence 1 + 4 = 5 pairs of bounds are computed). 
On the other hand, only 0.008% of the hypotheses re- 
quire between 100 and 999 refinement cycles. On aver- 
age, only 1.78 pairs of bounds are computed for each 
hypothesis (r = 1.78), instead of inspecting 16,384 pix- 
els for each hypothesis as in the naive approach. For 
convenience, these results are summarized in Table [T] 

Table 1 Estimation of the position of a known shape 'A' in 
a noiseless input image. 



|A| 


/it {pixels) 


at (pixels) 


T 


3 





0.82 


1.78 



Experiment 3. In the next experiment the hypothesis 
space is enlarged by considering not only integer trans- 
lations, but also scalings. These scalings changed the 
horizontal and vertical size of the prior (independently 
in each direction) by multiples of 2%. In other terms, 
the transformations used were of the form 



where t = [t x ,t y \ is an integer translation (i.e., t x ,t y G 
Z), Diag(s) is a diagonal matrix containing the vector 
of scaling factors s = [s^s^] in its diagonal, and s x ,s y 6 
{0.96,0.98,1,1.02,1.04}. 

To characterize the set of solutions A of this experi- 
ment, in addition to the quantities fit and at that were 
defined before, we also define the scaling bias, /x s , and 
the scaling standard deviation, o~ s , as 
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respectively, where are the scaling factors correspond- 
ing to the z-th hypothesis in the set A (the true scaling 
factors, for simplicity, are assumed to be s x — s y = 1). 

In this experiment the set of solutions consisted of 
the same three hypotheses found in Experiment 2, plus 
two more hypotheses that had a ±2% scaling error. The 
performance of the framework in this case, on the other 
hand, improved with respect to the previous experiment 
(note the r in Table [2]). These results (summarized in 
Table [2]) suggest that in addition to the translation part 
of the pose, the scale can also be estimated very accu- 
rately and efficiently. 

Table 2 Errors in the estimation of the position and scale 
of a known shape 'A' in a noiseless input image. 



|A| 


lit (pixels) 


at (pixels) 


MS (%) 


°s (96) 


T 


5 





0.633 





1.26 


1.17 



Experiment 4- The performance of the framework is 
obviously affected when the input image is corrupted 
by noise. To understand how it is affected, we run the 
proposed approach with the same hypothesis space de- 
fined in Experiment 2, but on images degraded by dif- 
ferent kinds of noise (for simplicity we add the noise 
directly to the BF corresponding to the input image, 
rather than to the input image itself). Three kinds of 



T(x) = Diag(s)x + 1, 



(52) 



noise have been considered (Fig. 13 1): 1) additive, zero 
mean, white Gaussian noise with standard deviation 
cr, denoted by A/"(0, a 2 ); salt and pepper noise, SV{P), 
produced by transforming, with probability P, the suc- 
cess rate p(x) of a pixel x into 1 — p(x); and structured 
noise, S(£), produced by transforming the success rate 
p(x) into 1 — p(x) for each pixel x in rows and colums 
that are multiples of I. When adding Gaussian noise to 
a BF some values end up outside the interval [0, 1] . In 
such cases we trim these values to the corresponding ex- 
treme of the interval. The results of these experiments 
are summarized in Table [31 
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Table 3 Effect of corrupting the BF corresponding to the 
input image with noise. 



Noise 


|A| 


lit(pixels) 


<Jt (pixels) 


T 


7V(0,0.15 2 ) 


12 


0.16 


1.41 


4.59 


7V(0,0.30 2 ) 


11 





1.34 


2.66 


7V(0,0.45 2 ) 


11 





1.34 


3.12 


5P(0.083) 


3 





0.81 


1.6 


5(24) 


6 


0.5 


1.08 


2.76 


SV (0.125) 


3 





0.81 


6.28 


5(16) 


3 





0.81 


5.49 


5"P(0.25) 


3 


0.47 


1.15 


42.77 


5(8) 


5 


0.4 


1.09 


39.17 



By comparing tables [T] and [3] we observe that the 
approach is relatively immune to the levels and kinds 
of noise applied, since /it and at did not increase much. 
However, the effects of the different kinds of noise are 
different. Adding moderate amounts of Gaussian noise, 
by making it uncertain whether each pixel in the image 
belongs to the foreground or background, increased by 
more than 3 times the size of the set of solutions, while 
doubling the amount of computation required (compare 
r in tables [T] and [3]). In contrast, moderate amounts of 
salt and pepper noise almost did not affect the set of so- 
lutions found, but they dramatically increased the nec- 
essary computation (because this kind of noise created 
many more "edges" in the image). To understand the 
effect of the position of the errors introduced by the 
noise, we adjusted the level of the structured noise I 
so that the same (approximate) number of pixels were 
affected by it and by the salt and pepper noise (i.e., 
approximately the same number of pixels were affected 
by S"P(0.083) and 5(24), by SP(0.125) and 5(16), and 
by SP(0.25) and S(8)). Because the structured noise 
concentrates the "errors" in some parts of the image, 
this kind of noise increased slightly the size of the set 
of solutions, but it did not have a consistent effect on 
the errors or the amount of computation required. 

Fig.[l3p-e show the shapes that were estimated from 
the noisy BFs in Fig. [T3ja. To understand how these 
shapes were estimated, recall that a discrete shape (given 



by (42)) is implicitly obtained while computing the 



lower bound of a hypothesis, and a semidiscrete shape 



(given by (46)) is implicitly obtained while computing 



the upper bound of a hypothesis. Thus, Fig. 13 i-e show 
the discrete and semidiscrete shapes obtained for the 
solution with the greatest upper bound. Notice how we 
were able to remove almost all the noise. It is also in- 
teresting to inspect the shapes estimated at an inter- 
mediate stage of the process, after only 50 refinement 
cycles of the bounds had been performed (Fig. [T3j3-c). 

Experiment 5. Another factor that affects the frame- 
work's performance is the "shape" of the prior. As men- 
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Fig. 13 (a) BFs corresponding to the input image after being 
degraded by three different kinds of noise (from left to right): 
Gaussian noise 7V(0, 0.45 2 ); Salt and pepper noise 5"P(0.25); 
and structured noise 5(8). (b-c) Discrete shape S (b), and 
semidiscrete shape S (c), estimated for the best solution af- 
ter 50 refinement cycles. Each shape was obtained for the 
corresponding input image depicted in the same column in 
(a), (d-e) Discrete (d) and semidiscrete (e) shapes obtained 
when termination conditions were achieved, for each of the 
input images in (a). The colors of the semidiscrete shapes at 
each partition element u> represent the fraction of the element 
(i.e., S(uj)/\u>\) that is covered by the shape (use the colorbar 
in Fig. [8] to interpret these colors). 



tioned before (in Section 6.3), the bounds computed for 



a partition element are tightest when either one of the 
BFs corresponding to the prior or the image are uniform 
in the element. This is because when one of the BFs is 
uniform, given the m-summary, there is no uncertainty 
about the location of the values of 5(x) in the element. 
Due to this fact, shapes that are uniform on elements of 
a coarser partition (referred below as "coarser" shapes) 
are processed faster than shapes that are uniform only 
on the elements of a finer partition (referred below as 
"finer" shapes). This statement is quantified in Table [4] 
which summarizes the results of five experiments. 

Each of these experiments is identical to Experiment 
2, except that a different shape was used instead of the 
shape 'A'. These shapes are depicted on the first col- 
umn of Table |4j Note that changing the shape in these 
experiments did not affect the set of solutions found (in 
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Table 4 Effect of the shape on the framework's performance. 



Shape 


|A| 


fit (pixels) 


<Jt (pixels) 


T 


■ 


1 








1.00 


• 


1 








1.04 


▲ 


1 








1.10 


* 


1 








1.50 


* 


1 








1.77 



all cases this set contains only the true solution and thus 
A*t = °t = 0), but it did affect the amount of computa- 
tion required in each case (see the column labeled 'r'). 
In particular, note that since the shapes are roughly 
sorted from "coarser" to "finer" (as defined above), the 
total computation required correspondingly increases. 
Interestingly, to estimate the pose of the first shape (a 
square of size 128 x 128), only one pair of bounds were 
computed (in constant time) per hypothesis (instead of 
processing the 16,384 pixels of the shape prior). 



7.2 Experiments on real data 

In this section we apply the proposed framework to the 
problem of estimating the class, pose, and segmentation 
of a shape (a letter in this case), for each image in a 
set of testing images. In order to do this, as before, we 
define multiple hypotheses and use the proposed frame- 
work to find the group of hypotheses that best explains 
each test image. 

More specifically, one hypothesis is defined for each 



transformation (described by (52 1) and for each class. 
To save memory, and since it was shown before that 
hypotheses that are far from the ground truth are eas- 
ily discarded, we only considered integer translations 
(around the ground truth) of up to 5 pixels in each 
direction (i.e., t x ,t y e {— 5, — 4, . . . , 5}), and scalings 
(around the ground truth) by multiples of 4% (i.e., 
s x , s y £ {0.92, 0.96, . . . , 1.08}). In contrast with the ex- 
periments presented in the previous section where only 
one class was considered (and hence there was no need 
to estimate it), in this section multiple classes are con- 
sidered and estimating the class of the shape is part of 
the problem. We consider 6 different priors for each let- 
ter of the English alphabet , giving a total of 26x6 = 156 
different priors (and classes). This results in an initial 
set of hypotheses containing 471,900 hypotheses (121 
translations x 25 scalings x 156 priors). 

To construct the priors for each letter, we com- 
piled a set of training shapes from the standard set 
of fonts in the Windows operating system. This set was 
pruned by discarding italicized fonts and other fonts 
that were considered outliers (e.g., wingdings). The re- 
maining shapes were then divided into six subsets us- 
ing A;-medoids clustering [IS], where the distance d(Si, 
S2) between two shapes Si and S2 was defined as the 
number of different pixels between the two shapes (i.e., 
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Fig. 14 The six shape priors (BFs) obtained for each of the 
classes 'A' and 'T. 

d(S 1 ,S 2 ) = \SiUS 2 \SinS 2 \) after "proper alignment" 
(this "alignment" is described below). It can be shown 
that in the case of 0/1 priors ("shapes"), this distance 
is equivalent to the evidence defined in (39). Then all 



the letters in each cluster were aligned with respect to 
the medoid of the cluster by finding the optimal trans- 
lation and scaling that minimizes the distance d men- 



tioned above, and (13) was used to compute the prior 
corresponding to the cluster. The six priors correspond- 
ing to the letters 'A' and 'T' are shown in Fig. [l4j The 
number of priors per class (six) was chosen to maxi- 
mize the classification performance Pi (defined below). 
We also tested having a variable number of clusters per 
class, but doing so did not improve the classification 
performance. The resulting priors used in the following 
experiments, as well as the set of training shapes used 
to construct them, can be downloaded from |30j . 

The testing images, on the other hand, were ob- 
tained by extracting rectangular regions (containing each 
a single letter) from old texts exhibiting decoloration 
and ink bleedthough, among other types of noise (see 
examples in Fig. [15] and in the images of these old texts 
in [30] V The corresponding BFs were obtained from 



these images by background subtraction using ( 12 ). The 
probability density functions (pdf's) p(f\q = 0) and 
p(f\q = 1) in that equation were learned from sets of 
pixels in the background and foreground that were man- 
ually selected for each image. In this case we learned a 
single pdf for the background, and a single pdf for the 
foreground (not a different pdf for each pixel). While 
more sophisticated methods could have been used to 
obtain these BFs, this is not the goal of this article, and 
in fact, noisy BFs are desirable to test the robustness of 
the approach. This testing set, containing 218 letters, 
is used to measure the classification performance of the 
proposed framework. It is important to note that the 
testing and training sets come from different sources 
(i.e., there is no overlap between the testing and train- 
ing sets). 

Since the proposed approach does not necessarily 
associate a single class to each testing image (because 
the set of solutions obtained for an image might con- 
tain solutions of different classes) , we report the perfor- 
mance of the framework with a slight modification of 
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Fig. 15 Sample letters in the testing set (top) and their cor- 
responding BFs (bottom), obtained by background subtrac- 
tion using ( |12[ ). Note the different types of noise present and 
the different typographies. 
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Fig. 16 Confusion matrices for /3 = (left), /3 = 0.5 (mid- 
dle), and f3 = 1 (right). Use the colorbar in Fig.|8]to interpret 
the colors. 

the traditional indicators. An element {i, j) in the con- 



fusion matrix Co (in Fig. 16 1 indicates the fraction of 



solutions of class j (among all the solutions) obtained 
for all testing images of class i. It is also of interest to 
know what the classification performance is when only 
the best solutions are considered. For this purpose we 
define the confusion matrix Cp (0 < j3 < 1) as before, 
but considering only the solutions whose upper bound 
is greater or equal than jp, where jp = L+/3(L—L) and 
L and L are the maximum lower and upper bounds, re- 
spectively, of any solution. Note that 70 = 7 = L when 
7 is defined as in Section [3j and hence all the solutions 
are considered, and that 71 = L and hence only the so- 
lution with the largest upper bound is considered. The 
confusion matrices Co. 5 and C\ are also shown in Fig. 

m 

Similarly, the total classification performance, Pp, is 
defined as the number of solutions of the correct class 
(accumulated over all the images in the testing set), di- 
vided by the total number of solutions. As above, the 
solutions considered are only those whose upper bound 
is greater or equal than jp. The total performance ob- 
tained in this case was P = 82.5%, P .5 = 86.5% and 
Pi = 90.4%. As expected, when only the best solu- 
tions are considered, the performance improves (i.e., 
Pi > P0.5 > Po). 

For completeness, Table [5] summarizes the average 
pose estimation "errors" obtained in this case. The quan- 
tities |A|, /Zjr, Wt, JTg, and ag in this table were respec- 
tively obtained as the average of the quantities |A|, /j, t , 
at, fig, and ag (defined before) over all images in the 
testing set. In contrast with the synthetic experiments 




Fig. 17 Examples of segmentations obtained in this experi- 
ment, (a) The original BFs computed from the input images, 
(b) Discrete shapes estimated for each of the BFs in (a), (c) 
Semidiscrete shapes estimated for each of the BFs in (a). As 
in Fig. |13| the colors in these semidiscrete shapes represent 
the fraction of each partition element that is "covered" by 
the shape. 

in this section the ground truth pose is not available. 
For this reason, and for lack of a universally trusted 
definition of the true pose (which is ill defined when 
the shapes to be aligned are different and unknown), in 
these experiments the ground truth pose was defined by 
the center (for the location) and the size (for the scale) 
of a manually defined bounding box around each letter. 
Therefore, the errors in Table [5] must be analyzed with 
caution, since it is not clear that this arbitrary defin- 
tion of the ground truth should be preferred over the 
solutions returned by the framework. In fact, in all the 



inspected cases (e.g., see Fig. 18 1, we did not observe a 
clear "misalignment." 

Table 5 Mean "errors" in the estimation of the position and 
scale of the unknown shapes in the images of the testing set. 
The mean is computed over all the 218 images in this set. 



|A| 


/it (pixels) 


at (pixels) 




<??(%) 


271 


1.36 


2.26 


2.70 


6.64 



Note in Table [5] that the average number of solutions 
per image (271) is only a small fraction (0.06%) of the 
total number of hypotheses (471,900). Moreover, these 
solutions are (in general) concentrated near the "ground 
truth" defined above (judging from the mean "errors" 
in the table). 

To illustrate the types of segmentations obtained by 



reported in Section 7.1 for the experiments reported 



the framework, Fig. 17 shows the shapes estimated for 
some of the images in the testing set. Note how most 
of the "noise" has been eliminated. 

In order to understand why misclassifications oc- 
curr, let us look at some solutions found by the pro- 
posed approach. Each column in Fig. [18] shows two so- 
lutions found for a single testing image. In the top row 
a solution of the correct class is depicted, while in the 
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Fig. 18 Representative examples of the solutions found by 
the proposed framework, given by their corresponding BFs. 
Solutions in the top row are of the correct class, while solu- 
tions in the bottom row are of a different class. These compos- 
ite images are produced by overlaying the BF corresponding 
to the testing image (in the blue channel) and the BF corre- 
sponding to the solution (in the red channel). 

bottom row a solution from a different class is depicted. 
It can be seen in the figure that when the fit of the cor- 
rect solution is not very good, solutions of an incorrect 
class cannot be discarded. This happens in the case of 
rotated or distorted letters (e.g., 'D' and 'N' in the fig- 
ure) and in cases where there is not a good fit in the 
database of priors (e.g., 'F' in the figure). These erros 
can possibly be overcome by considering richer trans- 
formations and/or more numerous priors, respectively 
(more on this in the next section). 



8 Conclusions 

This article presented a new type of algorithms, namely 
hypothesize-and-bound algorithms, to significantly trim 
down the amount of computation required to select the 
hypothesis that best explains an input (e.g., an im- 
age), from a set of predefined hypotheses. These algo- 
rithms are specifically targeted to perceptual problems 
in which the large size of the input imposes that these 
problems be solved by "paying attention" to a small 
fraction of the input only. 

These algorithms have two important properties: 1) 
they are guaranteed to find the set of optimal hypothe- 
ses; and 2) the total amount of computation required 
by them is decoupled from the "resolution" of the input, 
and only depends on the current problem. 

After describing the general paradigm of H&B algo- 
rithms, we instantiated this paradigm to the problem 
of simultaneously estimating the class, pose, and a de- 
noised version of a 2D shape in a 2D image, by incorpo- 
rating prior knowledge about the possible shapes that 
can be found in the image. In order to instantiate the 
paradigm, we developed a novel theory of shapes and 
shape priors that allowed us to develop the BM for this 
particular problem. 

We consider that the theory and algorithms pro- 
posed here are not just limited to solve the problem de- 
scribed in this article, but rather that they are general 
enough as to find application in many other domains. 



In fact, the application of the exact same FoAM (de- 
scribed in Section [3| and theory of shapes (described in 
Section |5| have already been instantiated to solve the 
problem of simultaneously estimating the class, pose, 
and a 3D reconstruction of an object from a single 2D 
image |3"2"] . 

Even though the contributions of this work are main- 
ly theoretical, we showed preliminary results of the pro- 
posed framework on a real problem. Nevertheless, we 
acknowledge that to provide state-of-the-art results the 
framework needs to be extended in several directions 
(which is beyond the scope of the current article). For 
example, a limitation of the current framework is in 
its ability to deal with classes of shapes that are too 
diverse. These classes result in priors with large parts 
whose probabilities are not close to or 1, and thus 
the bounds obtained are not tight, reducing the effi- 
cacy of the proposed method. These unspecific priors 
are obtained, for example, when the parts of the shapes 
in the class are not in consistent positions. To address 
this problem, in Section |7.2[ we divided highly vari- 
able classes into subclasses with less variability. This 
addressed the problem created by the highly variable 
classes, at the expense of creating many more hypothe- 
ses. 

Another direction to extend this work that addresses 
the growing number of hypotheses mentioned above, is 
to exploit the redundancy among these hypotheses. It 



was explained in Section 7.f that to prove that two (or 
more) hypotheses are indistinguishable, it is necessary 
to refine these hypotheses completely. By doing so, how- 
ever, the performance drops significantly. This problem 
gets exacerbated as the number of indistinguishable hy- 
potheses increases, which in turn happens when classes 
are divided in subclasses or when the number of degrees 
of freedom are increased. In order to exploit the redun- 
dancy among hypotheses, bounds can be computed not 
just for individual hypotheses but for groups of them, 
a la Branch & Bound. This new algorithm would then 
work by splitting not just the image domain (as H&B 
algorithms do), but also the space of hypotheses (as 
Branch & Bound algorithms do). This new algorithm 
is expected to be most effective when the hypotheses 
are very similar, which is exactly the case that the cur- 
rent approach is less efficient dealing with. □ 
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9 Supplementary material 

Lemma 1 (Properties of LCDFs and m-Summa- 

ries) Let LI be an arbitrary partition, and let S and 
D = {D u } ujen be respectively a semidiscrete shape and 
a LCDF in this partition. Then: 

1. Given a BF B such that B ~ D, it holds that 
(Fig. % 

r /"M 
sup / 5 B (x)5(x) dx = V / D-\a) da. ([29 ) 



2. Similarly, given a continuous shape S , such that 
S ~ S ', it holds that 



r _. rM 

sup / fe(x)5(x) dx = V / Dj^a) 



da. 



(301 



5. Moreover, for any a £ [0, |w|], i/ie integrals on 



the rhs of (|29|) and (|30|) can be bounded as 



/ D- X (a)da< 

«/ a 



J(a) (y/W 



7Ti— 1 



J=J(o) 



(311 



where J {a) is as defined in (22) 



^. XTie ras o/ (29) and (30) are maximum when 



S{uj) = — A u , thus 



da. 



(321 



Proof: 1. For simplicity, let us only consider the case 
in which the partition FI consists of a single element 
uj; the generalization of the proof to partitions contain- 
ing multiple elements is straightforward. Let us assume 
for the moment that D u (5) is strictly increasing and 
continuous, so that D u (D~ 1 (a)) = a Va £ [0, |w|]. 
Consider the continuous shape S* , defined as 

S*(x)^l(s B (x)-s) = ( 1 ' V6b(*)>5 (55) 
V / [ 0, otherwise, 

where 1 (•) is the Heaviside step function and 5 = D^ 1 
(\uj\ — S(uj)j. This shape is the solution to the Ihs of 



( 29 ) . To see this, notice that S* ~ S because 
S**(x) dx = |x £ uj : S B {x) > S\ 
= \u\-D u (I)=S(u). 



(56) 



Notice also that S* maximizes the integral on the Ihs of 



( 29 ) because it contains the parts of uj with the highest 
values of 5b- 



Thus, substituting (55) on the Ihs of (29), and defin 



ing the function g(x) = 5b{x) — 5, (29) reduces to 
S B (x)l (s B (x)-Sj dx = 
g(x)l (g(x)) dx + 5~ [ 1 (,g(x)) dx. (57) 



Using Lebesgue integration this last expression 
is equivalent to 



/•OO 

/ |{x £ uj : g(x) > 5}\ dS- 
Jo 

5\{x£oj:g(x)>0}\ = 



dS+ 



J |x £ uj : os(x) > 5 + <$} 

|x £ uj : (5 S (x) > <$} 



\w\-D u (6 + S)\ d5 + S [\u\ - D U {S)J = 

[\u\-D u {S)] dS + 5S(uj). (58) 



Now recall that the integral of a function f(x) can 
be written in terms of its inverse f~ 1 (y), if it exists, as 

MM 



a 



b /•/(&) 

" ' r\y) dy. 
/(<o 



f(x) dx = xf(x)\ a 



(59) 



Hence, it follows that ( 58 ) is equal to 



\uj\(6 

max u ) u rnax H'ti.r 
f D u (S max ) 

/ D- l {a) da + 5S(u) = 

\u\{5 max -$)- 5 max \uj\ + 5(\uj\ - S(uj))+ 
"M 

DZ l {a) da + 5S(uj) = 



\u\ 



D-\a) da, 



(60) 



as we wanted to prove. 

If D u {5) is not one-to-one (as we assumed above), 
the proof is essentially similar but complicated by the 
fact that S* can be chosen in more than one way (be- 
cause there are sets 8 C uj, with \8\ > 0, where 5b is 
constant) and this has to be handled explicitly. □ 

2. To prove 2 we proceed exactly as in 1, except that 
in this case we choose 5b to have its largest values in S, 
instead of choosing S to be supported where the largest 
values of 5b are. □ 
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3. From (21), it follows that 



/ D~\a) da<°^ J {a) da 
J a " m J Q 

/ J(a) da+ (j + 1) da 



j=J(a) ' 



(61) 



And because the integrands on the rhs of (61) are all 
constant, (31) follows. □ 



4- It is clear that each integral on the Ihs of (32) is 
maximum when the integration domain contains only 
the parts where D^^i. ) is positive or zero. Therefore, 



from (19), each integral is maximum when computed 



in the interval [Ab iUI , \u>\], which yields the rhs of (32), 
proving the lemma. □ 



Theorem 1 (Lower bound for L(H)) Let II be a 

partition, and let Yt = <Yt u > and Yu = \ Yh u > 

I ) uen I ' ) uen 

be the mean-summaries of two unknown BFs in II. 

Then, for any Bf ~ Yf and any B B ~ Yh, it holds 

that L(H) > L n (H), where 



L n (H) ±Z Bh + J2 4,(^0, 

and g* is a discrete shape in TI defined as 

> 



(40) 



(41) 



I 0, otherwise. 



(42) 



Proof: Since the set of continuous shapes q that are 
compatible with the discrete shape q is a subset of the 



set of all continuous shapes (from (lb)), it holds that 



L(H) > Z Bh + max 



sup / (<5 S/ (x) + S BH (pc)) g(x) dx 

(62) ' 

= Z Bh + max V* q(u>) / (5 B (x) + 5 Bh (x)) dx. 

(63) 



XTie terms in the integral in ( 63 I are, by definition 



the mean-summaries of the BFs (see (23),), thus 



L{H) > Z Bh + max £ + f ff 



It then follows that the discrete shape defined in ( 42 ) 



maximizes the rhs of (64), proving the theorem 



Theorem 2 (Upper bound for L(H)) Let FI be a 

partition, and let Yt — < Yt u > and Yjj = \ Yrj u > 

I ) uen I ' i uen 

be the m- summaries of two unknown BFs in TI. Let 

Yj(£ Hu (ui £ TI) be a vector of length 4m + 2 obtained 

by sorting the values in Yf >u] and Yh,u fin ascending 

order), keeping repeated values, i.e., 



Y, 



V 1 



■Y, 



4m+2 



/© H.u_ 

Sort Ascending I Yf_ u U Y B L 



(43) 



Then, for any Bf ~ Yf and any B B ~ Yh, it holds 
that L(H) < L n (H), where 



L n (H) ^Z Bh + J2 MH), and 



(44) 



uen 

4m+l 



j=2m+l 



(45) 



It also follows that the continuous shape that maximizes 



( 39 ) is equivalent to a semidiscrete shape q* in TI that 



satisfies 



VwSiT. (46) 



Proof: Let D f and D B be two arbitrary cumulative dis- 
tributions such that Df ~ Yf and D B ~ Y B ■ Then, 



sup sup / q 



B f ~D t 
B' H ~D H 



q Jn 



L(H) < Z Bh + 
?(x) ^(x) + 5 B /,(x)) dx, 



(65) 



and from (16), this expression is equal to 

= Z Bh + sup sup 
B' f ~D f SeS(n) 

B' H ~D H 

sup / (S b i (jc) + 5 B > (x)J g(x) dx. 



(66) 



Exchanging the order of the sup operations and us- 



ing (30), the rhs of (66) is less or equal than 



(64) 


Z Bh + sup 


sup 


sup / 






g~<3 


B' t ~D } Jn 



sup / £ B ' (x)g(x) dx 



2S 



Z Bh + sup 

qeS(U) 



Z Bh + Y, / ( Djl (a) + D H ] U (a) 

ue j7 q(.u) J\uj\-q(u>) 



H 



M-<?M 



D H.u>( a ) da 



da. 



Since Dj ~ and ~ Yjf , follows from (|21 
and (22) i/iai 



(67) 



J max 

m 



({ ' : ' / -, "} • {./:>',',...• a} 2,,/). 



which using ( 43 ) can 6e rewritten as 
^-(a)4^(|{ i: y/ ejrfU<a } 



2m 



(69) 



Therefore, it follows that 



r 4m+l + 1 



(j — 2m) da 



<. 4m+l 

m ' 



(i-2m)(y/g ff!W -y/ 0i?) J. (70) 



Substituting this last expression into (67 1 yields the 
bound 



L(H) < Z Bh 



rn 



sup 

fcG{l,...,4m+l} 



4m+l 



(71) 



^ (i-2m)(y/£ HiW -y; 0Hi( J. 



Since D ^ ff u ( a ) * s a non- decreasing, piecewise con- 
stant function that takes the zero value when a G ( 
g u , ^j^jj^]) it follows that the semidiscrete shape 
in (j46h maximizes ( 67 1 , and £/iat £/ie supremum in ( 71 1 



is obtained when k = 2m + 1, proving the theorem. □ 



